The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

All right, uh, since it's already like eleven twelve, I wanna start talking about the workshop. Um, so I'm gonna do this as a guided workshop, so I'll go through all the examples up here on screen, um, but you can also, uh, try and follow along. So the repo is there, the environment variables are there. Uh, so the environment variables give you access to a bunch of Foundry models that I've deployed on a Microsoft Foundry project. Um,

and they have a certain amount of token capacity. It is possible that we are gonna go over the token capacity, um, in, you know, in this session. Um, I only got-- found out very recently that this workshop was accepted, so I tried to get as much resources as I could in the small period of time. Um, so in that ENV file, there is actually, uh, multiple different possible environment variables that you can comment in or comment out if we do start getting four two nines, um, you know, uh, rate limit

exceeded errors. Uh, so yeah. So I'm hoping that there's enough capacity for all of you to be able to try out those models, and, uh, we will see. Uh, I also have my colleague Cedric here. Uh, so Cedric is going to be wandering around while I'm talking, so he might be able to help, uh, with, with issues as well. Uh, if you... In terms of setting up the code, if you already have Python and UV on your machine, then you can do a Git clone of the repo and run uv sync, and that should have everything you

need to run the examples, uh, once you have that .env. Uh, otherwise, you can make a GitHub Codespace that's, uh, basically a VS Code in the browser that has everything set up for you.

Who uses UV?

Uh, who here uses UV? Cedric wants to know. Okay. So it's not number one yet. But the other question here, who uses Python? All right, more people. Okay.

Pretty much everyone using Python uses UV.

Yeah. Oh, did you count the overlap? All right. Okay. All right. So if you didn't take a picture of that screen yet, just take a picture of it now. Otherwise, Cedric also has a picture of it on his phone, so he can go around if you need those links. Uh, so the goal of this workshop is to be able to get a feel for the differences between the different, you know, uh, top-of-the-line models and see how they differ and, you know, what you can improve about them. So I've set up a bunch of different examples and also evaluations and optimizations that we can do to improve our experience

across models. Uh, and I think this is a really important thing for all us to just be, to be doing as AI engineers because, you know, as we've seen, like, there's so many new models that come out all the time and, you know, the US government at any point could cut off access to a model. So we need to like, we need to have options, right? So we need to be able to s-- you know, point our AI applications at a different model and, and have it still work or know how to improve it. Uh, so that's the goal here is to get more experience with more models. So, uh, these are the models I've

deployed inside Microsoft Foundry. Um, so I've got, uh, GPT 5.5, that's a standard one I use all the time. But then I've got Kimi K2.6, that's a, a more recent one that's quite good. I've got DeepSeek V4 Flash, and then I have Mistral Large 3. So these are the four models that we'll be comparing in this workshop. Um, so as I was saying, you can... In order to set up the workshop, here's the repo that, that URL went to. So if you

don't have Python on your machine, you can just click on the Code button, and there'll be a big button here that says, "Create code space." So you click on that, it'll make a code space. So then here's the w-- uh, the code space, and then this has, um... I've already put the .env, but then you go and you grab that other, um, URL that I put up there to get the env that goes to this file here. You download it, you open that, and then you create a .env

with the contents. Okay? So this is pointing at a-- that Foundry project that has those four models deployed. Um, as I was saying, that does have, m- so, you know, it has token limits. Um, so if we start getting rate limits, you can start-- you can try these other models that also have the four product deploys. And then I have a separate one specifically for Anthropic just because it has different terms for that, um, project. All right. So, uh, so yeah, and then once you've got that set up,

then uv sync will sync all of the packages that you need for Python. All right, so that's the setup. Um, now I'm gonna actually go through it and do it, do it up here so that you can, you know, hopefully follow along if you want, or you can just watch, uh, watch what I do. Uh, so as I was saying, we do have the four models, and you can compare models just doing benchmarks. You're like, "Oh, you know, this one has good quality benchmarks and good safety benchmarks." Uh, you can

also look at, you know, the cost and throughput and stuff like that, and you can get this approximate notion for how good these models are. But you don't really know how good a model is until you use it for your particular scenario because there's so many different ways of using models, and a particular benchmark isn't really gonna tell you how it's gonna work for your scenario. So that's what we have in this repo here is a ton of different scenarios, and we can try out these models for each of them. So we're gonna start really, really basic. All right? So just

a single LLM call that asks the LLM to do something. So what you see for each of these files is that they all start off with the model variable, and they default to GPT 5.5. That's kind of our baseline. It's obviously, it's a very good model. Uh, but then if we wanna try a different model, we can just uncomment that line, and that will be the model that gets used. So this, uh, file here Is connecting using the OpenAI Python SDK to the Foundry URL and passing in the key from the environment,

and the prompt we're doing here is, how many times does the letter E appear in the following sentence? The elderly gentleman eagerly entered the elevator. Count carefully. All right. So then we send that to the OpenAI Responses API and say, "Here, use this model with this prompt," and then let's see the r- the output, right? Uh, so we're doing the classic thing of asking LLMs to count and so we'll see, you know, how well they're able to do. So I'm gonna say uv run examples, uh, with the name of this, this

file name here, and this is all in, you know, in the README. Um, so we're just going through one by one, so you can just copy and paste from there. All right. So GPT-5.5 is actually very good at math, surprisingly good at math. My colleagues have been running a lot of, um, finance benchmarks on it lately, and they're like, "Oh my God," like, "It does so much math in its head." So it does actually get this correct that there are 13 Es in this sentence, right? So now let's try, uh... Now we just trained it, right? So okay, let's say we want to try Kimi. So we go and we change to Kimi,

and now Kimi will be the model that's used, and we say, okay, can Kimi count as well as GPT-5.5? And it does. It gets 13. All right. Now you might be thinking, oh, was this just too easy? Let's try DeepSeek V4 Flash, and this time it gets six. Okay. So it missed out on seven of the Es, right? So and we can immediately see that there's definitely a difference between, um, you know, DeepSeek V4 and, um, those first two models. And then finally, we'll

try Mistral, and this one gets eight. Uh, so what we see, there's a difference here. All right. So when you see this and you're like, "Well, I want to be able to use all these models," then you start doing some prompt engineering and you're like, "Okay, what if I say, uh, you know, break the sentence into words, count per word, and sum. Show your reasoning." Right? Let's try, let's try that, right? And you know, we can fix our typos, not that it matters. All right, so let's try it with

Mistral and see, with a little bit of prompt engineering, can we get a better, better response here? All right, and it got it. Right? So, you know, by breaking it up, doing a little chain of thought here, it was able to come up with the correct answer, right? So immediately we can see that there's obviously differences across models, and if we change prompts, you know, we can get, uh, different outputs. All right. So now let's get more interesting and go to more real-world scenarios.

Uh, so let's look at RAG. So RAG is still one of the big uses. Arguably, really everything is RAG. Agents are RAG too. Um, so we have this... So RAG is when you ask an LLM to answer a question based off some context, right? And the thing that's important for RAG is that you want it to answer the question and stay grounded in the context. You don't want it to make up anything that is just from its weights, right? You're trying to force it to stick to the context, right? And that's what I find really varies across the models, is that the smaller models, the less

powerful models, are actually much more likely to go to their weights and pretend it was part of the context. Um, so what we have here is, um, we have a data set about carpenter bees, and we ask it a question, what are the key differencings, differences between how honeybees and carpenter bees build their nests? But this data set only has carpenter bee information, uh, so it should only actually be able to answer about carpenter bees. So a good response in this case would tell me the

carpenter bee information and then say, "Sorry, I can't tell you about honeybees. I don't know." Um, so we tell it, like, "Hey, your job is to answer based on the sources. Cite the sources. Um, here's your, here's your question. Here's your sources." All right, so let's go ahead and try that out and see what we get. Mm, mm, mm.

All right. And it's... Yeah, so and it's doing a search first, a very basic search. Okay, so it's sending... Did the search. All right, and what we see, and it says, "The data set does not provide details on honeybee nest construction." So this is what we actually want to see. This is from GPT-5.5. GPT-5.5 is very good at knowing what it doesn't know. So now let's try switching. Um, Kimi's also good as well, so I'm not gonna show every model, um, every time now just for, for sake of time.

Um, so let's try DeepSeek, right? 'Cause we've already seen that, you know, there's a, a distinction here. Now, DeepSeek once again says, "Provided sources do not include information." All right, so DeepSeek did a good job here. And now let's go to Mistral or Mistral. Sorry, Cedric, how do we say Mistral? It's French, right? French company.

Mistral.

Mistral. Cedric's gonna be my French translator. Every time we need to say it, you're just gonna say it. Now, Mistral-

Heads up.

Yeah, okay. Mistral. Um, so M- Mis- With Mistral,

uh, we do see that Mistral actually answered, uh, from its weights, uh, even though that information was not in the sources. So this is an example where a model went into its weights even though we didn't want it to do, right? Now, we can try to force it. Once again, we can try to do prompt engineering. Um, we can say, uh... Wait, actually, so it would say, uh, "If the, if the answer for a bit of info is not

in sources, say no source provided." I think this is what I got it to work. Uh, oh, and we'll try that. And do not make up an answer. All right, let's try this.

Uh, uh, uh. So we're trying to force it to realize when it's giving us information that's not in the weights, and sometimes if you force it to say, "Hey, for every bit of information that you're giving back in your answer, you have to prove that you got that from, from the sources," right? Um, so here I'm trying to have it associate each of the things in that, in that answer with a source and, um, and maybe that will force it to realize, oh, wait a second,

that wasn't actually, you know, in my, in my answers here. All right. It's still going now. It's gonna think. It's thinking really hard about it here. Okay, and there we go. Honeybee details are no source provided beyond interactions with carpenter bees . Um, so it definitely knew when things were not source provided. It still gave the information, so we got a little closer. But as you can see, this is definitely, uh, a difference that I see with RAG, and

RAG is what I work with the most. Uh, so it's, it's something you want to evaluate when you're looking at models to use with RAG is, are the models willing to admit what they don't know? How likely are they to make stuff up? All right, so now let's move to image input. A lot of these models are multimodal, so they will accept images as input, and, you know, there's lots of really cool, um, situations you can do with image input. So here we've got some files locally, and we're just gonna send them

to the LLMs and ask them some questions about the files, right? Um, I'll show you the files. Uh, examples, image input. Okay. All right. So the files we have here are this picture. This is an aurochs, not a unicorn. Uh, this picture, this is of some crocodiles that I saw in Costa Rica that I was very afraid of . Um, and then also we have, uh, this picture, which is a list of plants, uh, basically

like a screenshot of a table with some really small fonts in it. So we're asking each of the LLMs to interpret each of the images, right? So first we say, "Oh, is this a unicorn?" Right? That should be easy to say it's not a unicorn. Great. Then we say, "Hey, are these alligators or crocodiles?" That's a bit tricky. Uh, they are actually crocodiles because of their snout. Uh, I've spent a lot of time reading the Wikipedia page about crocodiles, and you look at their snout and then you know. Luckily, GPT-5.5 has als- also been on a Wikipedia binge, and so it

knows that when it sees that snout, uh, then that is crocodiles. And then the cheapest plant, it, it correctly found the cheapest plant in that, like, fairly small text screenshot of a table. So that's pretty impressive that, uh, GPT-5.5 could do that. Uh, so now let's try it with... Now, for these models, uh, we can't use DeepSeek 'cause DeepSeek only accepts text. It's not a multimodal model, so we're j- only gonna try Kimi and Mistral. Uh, so you know, it, it figured out

the aurochs. Okay, good. That's just making sure it works. All right. And then for crocodiles, it does say crocodiles correctly, and then cheapest plant, it got the cheapest plant. So Kimi is doing a great job as well. And then finally, let's go to Mistral here.

Let's see. All right. So, um, you know, it knows, it knows it's not a unicorn. It didn't know it was an aurochs. This one, it wasn't sure. It thinks they're alligators. Um, so it, you know, it's having a harder time looking at the image. It thinks that they're alligators. And then for the, the plants, it pulled out a plant, uh, that's $2.30, which is way more expensive than the cheapest plant. It was supposed to find one with fifty-eight cents. So the Mistral isn't able to answer these kind of images. Now, here's the thing. When it comes to multimodal

understanding, you can't really do much prompt engineering 'cause either, like, the im- like, the, the model either understands an image or it doesn't. Like, you might be able to do some prompt engineering if you know you're in a very particular domain, and there's some hints that you can give to it about how to read a particular kind of graph. Um, but really with image input, it's about figuring out, you know, for the kind of images that you're gonna send to the LLM, does the LLM understand those images? Is it going to, you know, is it going to work for that scenario, right? Um,

because it may be that, you know, maybe Mistral is really good at particular kinds of image tasks, but just not this, right? So it's all figuring out what is the task you wanna do, and is the model gonna do a good job at it? Now, what's interesting with the multimodal too is that a lot of these also depend on the language, right? Here I'm using English. Remember, all these models are super English-centric, but if you switch to a different language, like if I ask in Spanish if they're caimanes or, uh, cocodrilos, it actually thinks, uh, it says it, it gives the wrong answer. Because

it's getting so confused in its head between these different, like, words and how they map across English. Like, it just becomes too hard of a task to both understand the difference between alligators and crocodiles and also remember that caimán means alligator in Spanish and is not the same thing as the caiman that we have in, like, Florida. Anyway, it's like, it's fascinating to see that once we switch to Spanish, it fails at the task. Um, so that's another thing to keep in mind, especially we're in California. A lot of times when we develop applications here they are bilingual 'cause a lot, you know, fifty percent Spanish-speaking here. Um, so

you know, your, your mileage may vary as you change languages. Okay, so now let's go on to function calling, also known as tool calling. So tool calling is in fact the basis of, like, all agents. All agents are based off of tool calling, and this is a, a capability that the LLMs are trained with specifically during the training process. So they are trained to be able to look at a definition of a function or multiple function definitions and look at the user input and say, "Okay, based off that user

input, I wanna call this function with these arguments," and, um, and then, you know, just return that. So in this example here, I'm defining a function, uh, using JSON Schema. This is what the LLMs understand. So I say, "Okay, here's my function. It has the name create calendar event. It has this description, and these are the parameters it takes. It takes a title with a s- that's a string. Here's a description," right? And all of this is all gonna get sent to the LLM. So we send this full JSON

Schema to the LLM and say, "Hey, here's the function. You gotta decide whether you're gonna call this and how you're gonna call it," right? So this is in fact, even though this, this is like an object, this is prompt engineering here. Everything here is actually like me deciding what is gonna be the best for the LLM, right? What type should I use? What description should I have here? What should be the names of these parameters? 'Cause the LLM's going to decide based on all that information how it's going to call this function. Uh, so then we have a user message. So it's like, so the

user's like, "Oh, can you throw something on my calendar? It's a platform team sync, me, Sarah from Eng, Marcus, the new PM Priya, tomorrow at one thirty PM. It's virtual on Microsoft Teams." Okay? So then our-- then we tell the LLM, like, "Hey, you're a calendar assistant. Today is Monday, June twenty-ninth. Use the available tools to process the user's request," and we send it the tools. And so now the LLM is looking at the user request, looking at the tools, and in this case, it just has that create calendar event, and it's going to decide, "Hey, is there a tool here I can use to answer the user?"

Uh, and then we're just gonna go through and see what tool calls it decides to make. Uh, so there we go. So and, and this is, like, really low-level tool calling here. Uh, once we build an agent, we're gonna abstract on top of this, but I wanted to start with the most low-level tool calling we can do. And the LLM just responds back with, "Here's the name of the function, and here's the arguments that I want you to send." The LLM itself doesn't call the function. That's up to us to do. All right. So what we can see is that, um, I put my ex-- I put the

expected on the left, just so you can see, and we can see the actual. So GPT-5.5 did add me as an attendee, which I didn't really want, um, but it did pretty good otherwise. So let's try, uh, DeepSeek and see how DeepSeek does with this one. 'Cause what we wanna see, like when we're doing tool calling, we wanna see, is it calling the right tool? So it is in this case. It is calling the create calendar event tool. And is it passing in the arguments that we expect, right? We have certain arguments we expect. In this case, DeepSeek put you as

an attendee, and then for the location, it said virtual parentheses Microsoft Teams, which is not what I asked for, right? So it's not adhering to my argument, um, constraints as much. Let's try Mistral as well and see how it does. So once I see this, I can start to realize, like, okay, there's some ambiguity, right? It, you know, if it, if it called it with the wrong arguments, that means there's ambiguity. I need to go back into my descriptions and say, you know, uh, you know, "Don't, you know, don't include me or

you, just named attendees." Um, you know, you could say first and last if provided, right? So you start to see, you know, what y- what you haven't been specific enough about and start to clarify that ambiguity and then, you know, evaluate all the different, um, inputs and see how it's doing. All right. So that was a basic tool calling. The next thing we wanna do, we're building up to agents here. The next thing is to do tool calling in a loop, right? So when we have an agent, it's

calling tools in a loop until it reaches a goal. Um, so we're gonna demonstrate using this, just a function calculate. So this calculate takes a mathematical expression and returns back the output, right? And we have, you know, basic support for, uh, evaluating these, uh, mathematic expressions. So we're giving it basically a word problem. It's like, oh, the store is having a sale. If I buy f- three shirts and pay, you know, this much tax, what's my total? And this can be done several different ways. It can be done with multiple

tool calls to that calculator call with, like, one little expression at a time, like do this, and then, then this, and do this, and then this. Or if the LLM is very clever, it could batch all that into one long expression and do a single tool call. So here we're testing efficiency, right? Where, you know, there's mul-- like a lot of times once you have tool calling in a loop, the differences between the models is their efficiency. Are they able to get to the goal faster? Can they accomplish the same goal with less tool calls,

right? So that's what we're gonna say. Uh, the only system prompt is do not do math in your head. Okay. So let's see. Uh, that one's really for GPT-5.5 'cause it's so proud of its math abilities that it's always trying to do it in its head. It's like, come on. All right. So GPT-5.5 indeed does the entire calculation in one expression. So it only required one tool call to get to the final result. So that's good. That's good. That's, you know, fast. It saves us time.

So now let's try with, um, DeepSeek. The only reason I skip Kimi all the time is 'cause it's actually a very good model, and so it does pretty much o- on par with, uh, GPT-5.5, uh, for most of these. Okay. So now we can see with DeepSeek, it is going through, and it's doing... It's doing like so many... How many? It did like four. It did four tool calls, right? So it still got to the correct answer, but it took four tool calls, and every one of those tool

calls is another network request, right? And so, you know, when we're evaluating two, two models, we wanna both look at the final answer, but also how, you know, how long did it take to get to that final answer, right? Um, actually, Kimi did do something really interesting the other day. Let's see if it does it again, uh, with this one. 'Cause Kimi, most of the time, works quite well, but Kimi does sometimes get into weird loops, um, when it's kinda like double-checking itself. Uh, okay, this time it did

well. Yeah, the other day when I did Kimi, it, like, got the right answer on the first try and then decided to check its work with, like, four other calls. I was like, "No, you're good. You're good." Um, so that's what, that's what we're looking at here, okay, is how many, how many calls does it actually need to get to the end?

All right. So next, um, we can go to, uh, agent loops. Okay. I'm actually going quite fast through this now, actually. Uh, okay, good. Um, all right. So now we've got the basis, right? We've seen tool calling, tool calling in a loop. So now we can actually build like a proper agent, and for this, we'd usually use an agent framework. Uh, I've gone ahead and put three different agent frameworks in here, uh, so that you can pick your favorite. So we have LangChain, Pydantic AI,

and Agent Framework. Uh, I'm sure there's other agent frameworks as well that I missed, but hopefully you can find one that you like there. Uh, so these are all, you know, the same right here. All right, which one do we wanna show? Someone vote LangChain. All right. LangChain is still very popular, and, uh, hopefully some LangChain folks here today at the conference. All right, so with LangChain we are using LangChain v1, um, which is, uh, a nice, uh, new SDK version of theirs if you haven't tried it in the last year.

And so we connect to the Foundry models from LangChain, and what we have here is a travel planning agent, which is a classic example. So it has tools. So we have a tool for search flights, search hotels, and check budget to check if a certain flight and hotel fits within the budget, uh, and then search activities within a city. Uh, then I've got some middleware, which is just for me to s- see what tools are happening, and then we have the actual agent call. So the agent has a system prompt that says you're gonna help users find flights and hotels.

Uh, you're... You know, it has some rules here, like search for both flights and hotels, prefer direct flights, yada, yada, always check, check budget, and then ask for activities at the end based off the remaining budget. Then we have the user message, "I need to fly from SF to New York City next month for three nights. My total budget is thir- six-- uh, six hundred, and I want direct flights and an activity." All right. So we have all that, and then we are running the agent. There's some additional code here that's just logging out the tool calls so that we can see it. So let's

go ahead and try that out.

Okay. Mmm, mmm, mmm. And so this is going to log out the tool calls that happen. We're starting off with GPT-5.5 here. Now, what's interesting here, you'll see turn one... Let me make this bigger. Turn one, it says, is parallel. So one thing is-- you can do with LLMs is, um, is make-- Many LLMs support parallel tool calling, where if you give them a bunch of tool definitions, they can suggest multiple tools that can be called at the same time in the same turn. And this is really good

from a performance perspective because on your server you can actually file-- Like, you can send off requests in parallel to, like, two different databases, right? Um, so as long as they're independent of each other, this is a great thing that LLMs can do, is say, "Hey, these two, these two things can be done in parallel. Why don't you do them?" So, uh, GPT-5.5 on the first turn said, "Hey, search for hotels in New York City and search for flights in, uh, from SF to New York City," 'cause it's decided that there wasn't independency between them. And then once that came

back, once it had all the options, it did a check budget, um, based off of the flight cost and the hotel cost that it got back from that call. And then it saw what the remaining budget was, and then it did a call for search activities with that remaining budget. So this is probably the most efficient agent that we can have for this scenario. So let's see if we get similar, um, results when we change models here. Do, do, do.

And of course, here you could even change and compare across both models and frameworks, right? Because every framework has a slightly different tool calling loop. We also call it the harness, right? So a lot of times is you're-- you wanna compare both models and harnesses and, and you'll see there's interesting differences across them. All right, so this time we're with Kimi, and Kimi is not parallelizing, um, as well. I don't know if that's because Kimi doesn't support parallel tool calling or it's just deciding not to in this case. So this time it did the same... Actually,

it did even more call. So it did search flight and search hotels. It did over two turns, so that's already less efficient 'cause it didn't parallelize. Then it decided to do two calls to check budget 'cause I think it was checking, like, one more option, and then it did the final thing. So this one took two extra tool calls, and it took longer because it didn't parallelize that first, uh, set of turns. All right, let's see how we do with DeepSeek.

And you can start to see why it's hard, because everybody asks me, like, "How do you evaluate agents?" Agents are the hardest to evaluate because there's so much going on. There's so many different things that are happening along the way. It's like, are-- Is it calling the right tools? Is it calling them efficiently? Is it calling things in parallel? Um, in this case, you know, this one, uh, didn't do any parallelizing, um, but at least it only did four tool calls, so, you know, reasonably efficient. And then Mistral. Mistral often... We'll see. Mistral often

forgets the very last step. And so we'll see. Um, you know, that's the other problem with agents. Sometimes agents when they get to the... Like, once their context window fills up, they might forget some earlier instructions. They may not be as thorough, um, with, you know, paying attention to what the original steps were. And sometimes you have to, like, kind of prompt them along the way to, to remember, like, "Oh, wait, you're supposed to, you know, finish up with this final step here," right? Okay, so it did... It didn't do

parallel. So what we can see is GPT-5.5 is very good at parallel tool calling. Um, the other ones not, not-- aren't doing it. Okay. So Mistral decided to only search flights and search hotels and then not do anything else. Um, so it did, it did-- It kind of gave up there. Um, so that's an example where, okay, we'd have to do some ad-additional steering there to be like, "Hey, are you sure that you have everything, right?" Um, you could use something like the Ralph loop. Uh, Jeff Huntley is here today. He made the Ralph loop. And, you know, you can, you can force,

uh, you know, you can force agents to be more aggressive, uh, instead of just using the default loop. All right. Okay. So that, uh, that was an agent example. I was saying we also have agent examples in the other frameworks. Um, but what you see is the code is really similar across them. Uh, I tend to think most agent frameworks are, you know, pr- more similar than they are different at this point. Um, but, you know, you're welcome to check them out. All right, so now we

can talk about evaluations, right? So we've been doing everything just, you know, manually running them and observing the results. That's nice to get a feel for the kind of differences that you can see. But when you're actually gonna be, you know, doing something for production, you do want to actually have bulk automated evaluations that you can reliably run and compare results, right? So the first step that we can do is just, uh, doing evals for, um, with-

without, you know... just using string comparison, right? Um, so the example I have here is with that createCalendarEvent tool call 'cause what we can do is, like, we're actually getting back the arguments. We can actually just do a straight string comparison and say, "Hey, did it give back virtual?" Right? We wanted virtual for this. Did it give back virtual? So when you're testing a single tool call, you can just check to see, did it check... Did it ca- call the right tool, and did it use the right arguments? So that's what we're gonna do in this one. Um, and we set up

a bunch of test cases, right? 'Cause when you're running evals, you wanna have lots and lots of test cases. You also want it to be easy to add more test cases 'cause once your app is in production, you're gonna see how users really use it, and you're like, "Oh, okay. Thank you. Everything's broken." No, and then you're gonna go and add those new cases of user input to, uh, your, you know, your evaluations to make sure that you don't drift, right? This is called, like, m- uh, model drift or your, your, um, your drift away from it. So you don't want your evals to drift away from

your user input, so once they're in production you want a good system for getting new user inputs into your eval cases. So here what I do is I've got, like, lots of different, um, you know, uh, cases here. Like, here's a user message. Here's the checks, and these checks are just doing really string comparison, uh, some list comparison, and we check different things, right? We check the date math. Uh, I always check LLMs on date math 'cause usually they're kind of bad at it. So another tip, if you're doing dates, give them... just give them a tool to do date math, okay? 'Cause it's too hard to do date

math in your head, uh, especially on leap years. I was evaluating an LLM on a leap year. That was hard. Uh, then we had, um, okay, ambiguous durations. I did one in Spanish, right? It's nice to test across different languages, whichever languages you're gonna support. Uh, this one I guess is written, written real slangy. "Yo, can you book the big conf room?" Okay. All right, so we have various edge cases there, right? And then we're-- I'm gonna run these evals. M- m- mm.

Okay, and these evals, in this case, I'm gonna run it across each of the models and the use cases. A lot of times it... in reality when I'm running evals, I'm just running it on a single model, and then I'm comparing to, like, a previous run where maybe I'd done a different model. Uh, but, you know, for today, uh, since we are actively comparing across models, I'm going to run it across all the models. So we can see, um, it's... for each of them, it's doing the tool call, checking the args that come back, and deciding if it's a pass or a fail.

Uh, now here's the thing about evals is that you don't wanna just have these, like, just be part of, like, your pytest, like your test suite, because these are slow. These are expensive. Yeah, question.

Um, I see you've just raw, you know, checking strings against each other. Uh, do you know of any frameworks to do this a bit more elegantly? I know- ...

You don't like my string checks. Um, yeah, so framework-wise for evals, a lot of... so, so for... most observability providers have evals built in, right? So if you're doing LangFuze, LangSmith, Lang- Lang- LangChain, um, Arize, like, they all have their own framework built in. For Foundry, we have, um, we have two different frameworks, uh, for it, and I actually... I do have the examples in the repo, um, of those ones. There's Azure AI Evaluation SDK, Open, uh, AI.Evals, um, from the OpenAI package. So there's a bunch of frameworks. There's also

DeepEval and-

Vally too.

Prompt, PromptFu in JavaScript if you're a JavaScript person.

Vally too.

In which one?

Vally.

Vally?

It's spelled Vally.

Mark it V- Vally. Okay.

V-A-L-L-Y.

V-A-L-L-Y. Okay, I haven't even used that one. Um, so there's a number of eval frameworks. Um, I'll... I'm gonna use a fancier one soon. But for a string check... But here's the thing, like, uh, like, for string check you don't really have to do a fancy one, um, uh, because you're, like-

If you're looking for it to do a specific-

Yeah

... to do the parallelization like you mentioned before, like-

Yeah, yeah. So for tool calls, yeah, that's where you're gonna get a bit fancier. Yeah. Yeah, so I wanted to start with the basic. But yeah, most of the time we can't get away with just the basic, right? Um, oh, they did actually really well this time. Wow. Nice. Um, so usually we're going to have to move to fancier evaluation. Um, so that's when we start to bring in LLM-as-a-judge. Uh, so this is... 'Cause a lot of times, you know, the, the output is a lot fuzzier, and we need to use an LLM to help us reason about that

fuzziness, and there's many frameworks that do wrap this. Um, for this example, I just decided to write it entirely in the, in the code so you could see. So the idea when you use an LLM as a judge is that you need to give it... you know, put the LLM in judge mode and say, "Okay, you are an evaluator. This is what you're evaluating, and you need to either say does it pass or does it fail," right? We're trying to force the LLM to say, "Does this pass or fail?" Um, and we also want it to give a reason,

right? It's really, really helpful to have it give a reason. So we want pass, fail, and a reason. And then we give it the query. We give it the response. And for groundedness, we also wanna give it the context 'cause that helps it actually reason about it. So sometimes in order to really evaluate, you do need access to more information. So like tool calls, if you're trying to evaluate whether it was an efficient user of tool calls, you can't... You have to give it everything. You have to give it the tool call definitions. You have to give it the traces of the tool calls it made, right? So to build really good evaluators, a lot of times you're, you're not just giving it the input

output, you're giving it every- everything that happened along the way, ev- everything that was shown. Um, so here we, you know, we t-tell the LLM, "Hey, our, uh... We're gonna use the judge model of GPT-5.5." It's a pretty good judge model, and we're gonna ask for it to output some JSON that we can look at. So let me go ahead and run, uh, this, uh, LLM-as-a-judge. Now, here's the thing, um, this is going to be even slower because now each time we're running an eval,

we're both running our own code, which is using a model, and we're then using another LLM to judge that code, right? So every single eval case is actually two LLM calls. And then many times, you actually run multiple LLMs as a judge. Like, you'll have, like, five different evaluators that you're running. Each of those is gonna be a separate LLM call, right? Um, so that, that's what is tricky about evals is that you can't just run them all the time. You can't necessarily run them on every single CI, uh, and every single commit

because they are fairly slow to run. 'Cause normally in production, like for RAG, when I'm evaluating RAG, you w-want-- usually want like 200 test cases at least, maybe 800, and that's li- that's like a lot of calls to an LLM, right? So that, that takes some amount of time. Um, so you do wanna think about how often you're gonna be, you know, running eval. So you can see that it's going through, it's text... It, it's got a bunch of different test cases, different questions and answers, and it's, it's looking at the results. Now in this case, what I did was that

I, like, pre-ran the, um, the questions and answers. This is a common way of doing it too. Like get, you know, get all your... Get your context for the evaluator beforehand, um, because then you can change your evaluator independently from your app, right? Um, and that's more efficient. 'Cause a lot of times what you realize is that your evaluator is incorrect, right? Like that you haven't written a good evaluator. Uh, and so that's what you have to figure out. When you're doing LLM-as-a-judge, you have to figure out like, "Wait, is the problem my app or is the

problem my judge?" Right? And, uh, you also wanna calibrate your judges with human input as well. So it definitely helps to be working on a domain you know or work with a domain expert that can help you label your data and say... And, you know, do they agree with your LLM-as-a-judge? Was it actually a pass or a fail? Uh, so here we get the results back, and we can see, you know, the model, um, and their scores. Um, most of them got like two out of three. Mistral got one out of three.

You know, so they're, they're... They have some work to do on being more grounded here.

So that is the generator of using LLM-as-a-judge. As I was saying, there's quite a few frameworks that, uh, make it, you know, that have built-in LLM-as-a-judge, that make it easier to do it. Um, but I wanna show that you, you know, you can, you can also do it yourself. Oh, also I should recommend, if you're interested in evaluation, hamill.dev. I don't know if he's at the conference this week, but he has the most amazing resources, and he teaches courses all the time about evals. And also, he has very good memes. I actually used these last night. So if you're ever chatting with someone

about evals and you need an eval meme, uh, th-what was what I was doing last night? Yeah, building an LLM-as-a-judge, it's, is hard.

Uh, so very important resource here. How can we trust the LLM judge? You have to calibrate your LLM judge. Okay. All right. So that was LLM-as-a-judge. So now, um... But then the, as I was saying, it is hard to build judge evaluators, right? Like, you have to like follow Hamel on Twitter and take his course and learn all these best practices for how you write, uh, a good eval. Um, uh, so we have this new tool from Microsoft

Research. It's very new, like the last week. Um, it's called Assert, and Assert helps you write the judges based off of a spec. So you write a spec. You say, "Hey, this is the agent I'm building, and this is, you know, roughly what I want it to be doing, my idea about how it should work and, um, what kind of users it should support." And then it will generate lots of test cases. It'll like flesh out your, your spec into, um, into actual judging rubrics and figure all of

that out for you based off of, you know, all the, the... You know, it's very based in, in research, right? So we've done that for the travel planner already in here. Um, so here's the... Um, so it's under this assert eval. Okay. So what we can see is that we write a YAML file 'cause we all wanted more YAML in our life. What I realized about YAML is that I might hate YAML, but it's also better than everything else for what I need it for. So there we go. Um, so in this YAML, you know, we describe,

uh, the context, and we describe, okay, we're gonna use this judge model. Um, these are the kind of dimensions we care about is budget, you know, budget pressure. So we c- we can give it more information if we know, like, oh, we care about ample budget, tight budget, infeasible. Um, you know, so you can, you can stratify it. This is called stratification. You can, you can pre-stratify if you want, uh, or you can just have it come up with the stratification. Uh, and then we have more dimensions here. And there's this, there's also a, a skill, an

agent skill that helps you write this YAML file so that you don't have to write it yourself 'cause no one wants to write it, right? Um, and then we have the actual thing that's evaluating. This is the LangChain travel planner that we saw before. Uh, so it's just making sure that it's tracing everything. 'Cause as I was saying, when you are doing evaluations, you do wanna see every single tool call so that you can really properly do these evaluations. All right. So then it, it runs it all, and then we can see the results. It does take a, you know, a decent amount of time. So we've already got the, um, evals in

here. Uh, so I've got them running. Uh, there's like a little like front end that you can use to view them. Okay. All right. So here is the result of this, um, a- Assert evaluation. Um, so basically it, you know, it came up with the different behavior categories, right? So making... Like budget infeas- infeasibility, direct flight preference, rule violation, required tool use, right? So,

uh, it comes up with what behaviors are pro-- you know, allowed or not allowed. And then it comes up with a test set. The test set is really f- is pretty fun. Um, and here we go. Scenario 65. Red eye before the installer walkthrough. The coach with six parent text one weekend to see Ama. I really like that these are obviously LLM-generated titles, right? So it came up with all of these... Coach with a clipboard. Um, it ca- it comes up with all these scenarios based off that, that YAML to try

and simulate users, right? 'Cause the whole thing, it's really hard to bootstrap your evaluation before you go into production, but you wanna have evals before you go into production, right? So how do you figure out, you know, what your test set is going to be? And so this ASSERT framework helps a lot with making that test set, uh, and bootstrapping your evals. Uh, so it has prompts and scenarios, and then we have the results. Okay, so for the results, it ran it, um, across DeepSeek, Kimi, and GPT-5.5 and, uh, came up with the, you know,

the numbers here. Um, and in this case, the, the lower numbers are better. And so we can see that, uh, GPT-5.5 did way better than the other ones. I was actually surprised 'cause Kimi, we've seen that Kimi has generally been quite good today, um, and I was thinking it was kinda on par. But at least for this scenario, uh, GPT-5.5 had the lowest number of violations, and so it- it's, it's scoring the best here. Um, so, so yeah. So this is

ASSERT. I will say I'm, I'm brand new to ASSERT since it only came out, like, a week ago. Um, so this is all I've, I've used it so far. But the idea here is that it's helping you to build those evaluators, to build that test set, and to follow the best practices and bootstrap your evals so that you can start, start off, um, you know, with, with some really good evals. 'Cause what we're seeing is that we keep telling people, like, "You gotta run evals. You gotta run evals," and then no one's running evals, or they're running, like, the default evals, like coherence

and fluency. Never run those. Like, LLMs are great at being coherent. You never have to check if an LLM is coherent. If some- if they're not, like, something really bad has happened. Um, I do sometimes use the groundedness and relevance evaluators for my RAG things, but most of the times, the built-in evaluators are not what you want. You want to build custom evaluators, and that's what this is helping you with, is like, okay, you need custom evaluators to really see if your agent is doing a good job, uh, but, you know, you need some help

bootstrapping those custom evaluators. All right. Oh, we have three minutes left. I've been blabbing. Okay. Finally, um, you know, earlier we were talking about prompt engineering. Like, okay, how can we tweak this prompt in order to work better for these models? That's pretty painful, right? If you have to tweak the prompt every time you switch models and ru- like, rerun evals, that's, like, a lot of work. Oh, there's a little timer. Thank you. Okay. Uh, so if you... Yeah, if you have to tweak the model every time, uh, that's

a lot of work, and you don't know which way to twe- tweak the model, right? You don't know the details of every single model, what's gonna really work well for it. So, um, what I've added in this as a final example is using DSPy. Has anyone used DSPy here? Cool. A couple people. So DSPy, the idea of DSPy is that you declare your program. Like, you say, "Hey, I'm making this program that's expecting to take user input and then turning it into a structured calendar event." So you declare the constraints of your program, and then

DSPy is going to figure out the prompt that adheres to those constraints. So DSPy is gonna write your prompt. So you're, you're basically checking your spec in and then having DSPy write the prompt based off of doing evaluations and saying, "Hey, what prompt is going to satisfy those constraints?" And it actually uses, uh, these, this cool, uh, algorithm called GEPA, Genetic Pareto Algorithms. It can use these genetic algorithms in order to come up with the next best prompt to try, which is really, really cool to watch it do that.

Um, so that's the idea of DSPy. And so basically, we have to set... So it's very different, right? You can't just take your standard code and just use it. You kind of have to rewrite your whole code, 'cause now, instead of checking your pro- like, your prompt isn't in your code. Your constraints are in your code. Your spec is your code. The prompt is an output from this DSPy process, right? So here, I'm doing-- I'm asking it to generate a numbered list where each line has an exact word count and the first letter of the line spelled a target word. So this is the kind of input, and then we have a bunch of constraints that are checking to

see, is the result correct, right? So you need to set up your spec and then set up, like, checks, tests to see, hey, does the output satisfy this constraint? And that way, DSPy can keep iterating hill climbing, right, so basically hill climbing, until it finds a prompt that satisfies those constraints, right? So you give it the student model, the reflection model, uh, then it runs this GEPA o-optimizer, which, you know, tries to find the best, um, prompt, and then we get back the result. So the result, I have it stored here,

so we can see it. It does take a fair amount of time. It took, like, six minutes on this task. Um, but here, this result is the actual instructions. So let me add word wrap into there. Right. So this is the system prompt that it came up with based off of those constraints and, uh, those, you know, th- those checks, right? And so this is nice because this is a system prompt that works with Mistral, 'cause I, I picked Mistral 'cause that was the one we were having the hardest time with, right? So especially with using smaller

models, this means we can use the cheapest model and still get the results we need, right? So there's been some great talks, um, by, like, Dropbox, Shopify, about how they've used DSPy plus GEPA so that they can move off the larger models and use the smaller models, like maybe even deployed on their infrastructure, and get the same results. But the key was trying to, you know, using this in order to figure out what that optimal prompt is. So this is a definitely a different way of writing your apps, and it, it can feel a little different to, you know,

to have the prompt be the output of this process. But it's also potentially a really powerful, uh, uh, way of writing it and can give you a lot more model independence and more flexibility to be able to use more of these models. Uh, so I wanted to, yeah, make sure that people knew about that as an option as well. Okay. All right. So, uh, so yeah, so that was all, all the things that I wanted to show. Uh, as you can see, there's lots of differences between models, different ways you can

improve the model output, and then evaluate it, and then improve it. Um, I do encourage you, like, yeah, just, just ch- tinker with the models. It's... I always learn so much whenever I, you know, try out different models, um, and see how different they are. Oh, I never showed Anthropic. I meant to show Anthropic. Okay. I do have some Anthropics deployed as well. With Anthropic, you do have to use a different API. That's why I wasn't using Anthropic most of the time. Um, but of course, you might be interested in Anthropic. They, they use the Messages

API instead, so you do have to switch your code around. Um, but I actually specifically like it for RAG because, um, here's the interesting thing, is that with Anthropic, they actually have this notion of citations built into the models. So you can actually... You provide the sources, uh, in this structured format. So you say, "Hey, this is my source. It's a document. It's text. Here's the text of it." And then you enable citations. And then the Anthropic models are actually trained to return back blocks of

text with citations, so you can extract those in a programmatic way, which is really nice when you're, like, building front ends. You can just link your citations. Um, so that's really cool. So that's even another thing, um, to look at, is, um, you know, whole different ways of calling models too. Okay. All right. Uh, and yeah, we're kind of at time here. Um, the, the repo will stay up, so you're welcome to keep working on the repo. I do need to rotate the keys

that I put in that .env because, I mean, I trust you, but I don't know how much I trust you, and Cedric alw- already said I was bad for doing that. Um, so I, I won't be able to keep those keys up for much longer. Um, but hopefully with the repo you can try it out. You can also try it out with Ollama. I added Ollama support yesterday, so if you want, you know, do wanna try Ollama too, if you're doing local models, you basically just change the examples to point at the local Ollama server, and, uh, you just have to change the model names as well, right?

Um, but generally they should all work with, uh, Ollama as well because it's same API. All right. Thank you, everyone. Oh, and there's stickers.