Running AI Application in Minutes: Quick Start with AI Templates

[upbeat music] Thank you so much for coming to the workshop.

My name is Gabriela de Queiroz, and I'm director of AI at Microsoft. I have Pamela here.

I'm Pamela, and I'm a Python cloud advocate. [laughs] So well done on those of you who said Python. Um, but, uh, I also, I worked in JavaScript before then for quite a long time, and I generally like lots of languages.

And then we got Harold. Here we go.

Hi. I'm Harald. I'm a PM on VS Code and GitHub Copilot Chat, so...

Awesome. Um, so today we are going to be t- talking or showing you how to run a AI application in minutes. So we are going to have a lot of, like, hands-on, so be ready to do, like, some coding.

Not coding, but, like, h- going through some coding, uh, using different tools, uh, GitHub Codespaces, Azure, and, and other tools that we are going to be talking about. But, uh, just to give a, a, a overview of, like, the agenda, I'm going to be talking about Microsoft for Startups a little bit, some of the partnerships, some of

the pain points, and then we go through the AI templates and hands-on. Uh, so Microsoft has a program for startups. So if you have an idea, if you have a startup, uh, you can apply to this program.

And what I always tell people is you don't have to have a startup per se, but if you have an idea, that's enough to apply for this program. And you get a lot of benefits, and benefits that can be, um...

I'll just skip. Uh, it can be, like, credits. So you get up to $150,000 in Azure credits. You also have third-party benefits, like a lot of, like, different tools that you can use.

And then of course, GitHub, Microsoft 365, LinkedIn Premium, and more. Uh, you can use all the different models from OpenAI, but also, like, Llama, uh, models from Cohere, uh, Mistral, and so on.

And the, the piece that I like the most is about the sessions, that you can get one-on-one sessions with people like me or Pamela, uh, that, uh, we volunteer our time to share our knowledge with founders.

Uh, we can talk about maybe, like, I don't know, you are hiring, and then I'm an expert in hiring, so you come and talk to me, and I say, "Hey, these are some of the best practice for you when you are building your team."

Or you can go to technical sessions and ask more, like, technical pieces, um, as well. And inside this platform, we have, like, several things other than the benefits that I mentioned and the guidance that I just mentioned.

It's what we call Build with AI. And inside, we have some AI templates that the idea is, like, you, you, you... we can help you accelerate, um, the, the AI application piece with some kind of, like, skeleton in a way.

Um, so, so you have something up and running in, like, f- few minutes.

Um, so again, you get cr- cloud credits. You have access to dev tools. You have the AI templates. You have the one-one, one-on-one guidance. Um, and no matter where you are in your journey, if you have an idea, if you are already building or if you're scaling, this program is for you.

Um, you have access to all the cutting-edge AI tools so you can innovate and streamline your AI development. And on top of, like, the Founders Hub, this program that we have, we also have, like, different programs, um, that it kind of like, it's like the next step.

Like, let's say you are now scaling, growing, and then you use all the credits. What is next? There is a next. Like, you know, we try to guide you through the whole process.

So there is something called the Pegasus program, um, where we help you to co-sell, go to market, and so on. And then there are some, like, strategic VC partners and, like, accelerators that we partner with.

So we have partnership with Y Combinator, Neo, The Alchemist, uh, et cetera.

Um, pain points for startups, there are a bunch of them. Uh, one of them is, like, you don't have time. You cannot wait to go to market. You have to go r- like, as fast as you can.

You have a lot of, like, resource constraints. We have some issues with scalability. You have... You don't have the support and guidance, and that's where we are trying to, um, help you with.

So now we are going to go to the fun part. It's, like, the AI templates. So that's where Pamela is going to show you all the amazing things that you can do with all the different tools.

All right. So our goal today is potentially having you deploy maybe even three different templates.

Three?

Okay. [laughs] Um, uh, so we have three different ones. I'll, like, just, um, you know, show in the, in the browser which ones we're gonna be deploying, right? So we have starting, we're gonna start simple with this chat application here, just to make sure everything's up and working.

And then we've got two different RAG applications. One of them is a RAG on a Postgres database, like RAG on a Postgres table that does, uh, SQL filter building.

And then we have RAG on a unstructured document. So here I've got a RAG on my personal blog [laughs] or, like, a RAG on, you know, internal company documents, whatever it is that you're gonna, whatever kind of documents you're gonna RAG on.

So those are the three templates we're gonna be looking at today, and we have it all set up so that you should be able to deploy those templates without spending any of your own money, and doing it all through our credits, which is yay.

All right. So- Um, the first thing you need to do is get this URL. So everybody open this URL on your computer. So it's aka.ms/aie-workshop. It should open up a, a Word document in the browser that looks like the screenshot you see here.

So you can either type in the URL or scan that QR code and get that open on your machine.

So let's make sure everyone's got it open.

Welcome, welcome. So go ahead, and once you've got your computer ready, put this, uh, put this URL in your browser. Harald, maybe you can just memorize it and then help anyone who doesn't have it.

Yeah, aie-workshop. [laughs] Uh, okay. So then let me go to the actual doc here.

So the first thing you need is a GitHub account. Does anybody here not have a GitHub account?

Okay, so everyone here has a GitHub account, great. If you don't have a GitHub account, you can sign up for one for free right now, and um, and that should be fine.

Um, the next thing you need is an Azure Pass. So this is something that we've got for this workshop, for this conference, and this is gonna let you deploy stuff on Azure without spending any of your own money.

So we got passes for 50 bucks, and they're valid for seven days, so if you do wanna keep hacking after the workshop, you can keep using your pass and, uh, after seven days it'll disappear just like Cinderella and the pumpkin.

Uh, so in order to get that Azure Pass, you do need to have some sort of Microsoft account. So you can use your, like, uh, you could use a personal Microsoft account if you have one.

Uh, so if you're, if y- like, how do you tell which one you're l- logged into right now? I guess if you just go to outlook.office.com, maybe you would know what Microsoft account you're currently logged into.

Um, and then you can see. Some people in the last workshop were, like, logged into their kids' Minecraft account, so [laughs] just, uh, just you d- you need a Microsoft account, and you might wanna double-check to see which one you're currently signed into if you are signed into a Microsoft account.

If you don't have a Microsoft account, no big deal. You can make one on the spot. I made one this morning. So, uh, if you do need to make one, you can just make up a new Outlook address and set it up that way.

Um, so you can also make it as part of this progress. So we're gonna go to this AZ check-in URL, and that's linked from this doc here. So if you don't have this doc, if you just came in, we can help you get this doc open so we can get this URL.

And, uh, we're gonna spend 10 minutes making sure we get through this step since it can be a little, uh, a little tricky. So when you go to this check-in URL, right, we put this in the browser.

It loads. This is what you're gonna see, and it says I can either create a GitHub account or log in with GitHub. So I'm gonna log in with GitHub 'cause I already have a GitHub account and I'm logged into this browser with it already.

So I'm just gonna click on that. And so what that's gonna do is create a pass for my GitHub account. And so we get a pass, so each of us will get a different code based off our GitHub account.

So this is my, you know, ba- basically my Azure Pass promo code, so I can copy that, and then there's this button here that says Get on board with Azure.

This is the next step, is to click this.

And then we get this screen which says, okay, this is, you can start.

And when I click this here, it says what my currently logged-in account is, so this is where you should check to make sure you're happy with what account you're logged in with and you don't wanna switch.

Um, I don't recommend using a corporate account. If you do have a corporate account, like, don't... just don't use it. It's gonna be problematic [laughs] for various reasons, 'cause corporate accounts may have restrictions that won't let you deploy things.

So we do recommend using some sort of personal account or making up a new account, so that's why you see I'm using my Gmail instead of my Microsoft. Uh, so I'll confirm my account, and then I can enter the promo code, and that was from this screen, so I still have this screen open, so I just go

there. I paste it in, and then we go S, uh, six, X, Y, Y, K. I think it's case insensitive.

Submit. And then it's, uh, gonna actually fail for me because I've already set this up on, on this, uh, thing here. Um, and this, if you see this, it's because you've already actually gone through this stage.

Uh, so for you, it should work the first time. And then, uh, it'll create the Azure account for you. And if it works, then what we can do is go to portal.azure.com, so portal.azure.com.

And we'll see what it, how it loads in.

Does a bunch of redirects. And then we can click on Subscriptions, and what we should see is there should be at least one subscription that says Azure Pass - Sponsorship.

So that's our key that we have done this correctly, and as long as we use this subscription when we're doing our deploys, we will not get charged any money.

Well, Microsoft will, but you won't. That's the important part. Okay, so we're gonna spend 10 minutes to make sure that we can get everyone through this, this stage so that we're all on the same page going forward.

So if you already got it, that's awesome. You can, um, you know, like, look at Harald's, like, uh, Facebook profile or something. [laughs]

So once you have that set up, the next step is the proxy

Um, so I'll just show that, uh, so that you can start playing with that. Uh, so here's- It's the next link in here. So the reason we have a proxy is because normally when you're using Azure OpenAI, you actually have to fill out a form and say how you're gonna use Azure OpenAI, and then somebody says, "Oh,

okay, yeah, that's a good use of OpenAI," because Microsoft doesn't want people to use AI willy-nilly. So we, you know, check to make sure that something adheres to our responsible AI principles.

Uh, we don't have enough time for you to go through that process while we're in a workshop, so we've set up a- an Azure OpenAI proxy that you can use during the workshop with the repos, and we have special instructions for how you can use this proxy with the repos since you can't use the actual Azure OpenAI.

Uh, so this- you can follow the link from the doc and log in with your GitHub account. Uh, I'll log out so I can show that.

Log in with GitHub. Okay, and it says I'm logged in.

And then we have an API key and a proxy endpoint, and that's all we need to be able to, uh, to use an Azure OpenAI instance. Now, normally I don't like to use keys, and I tell everybody to avoid them, but, uh, in this situation we are gonna be using keys.

And, uh, yeah, and these keys will expire at a certain point, so we don't have to worry about them being exposed. Uh, typically with keys we'd have to protect them very fiercely so that nobody was using them.

So you can go ahead and log into this and see your registration details, and then you can even play around with the playground. This is really similar to the Azure OpenAI playground or the openai.com playground if any of you played around with this.

Uh, you can see here you can play with the system message. That's how you like say like, "Oh, you're an AI assistant that constantly makes pirate jokes."

Yar. Uh, and then we update the system message. [laughs]

Oh, private. [laughs] I wonder what it'll do. [laughs] Uh, there we go. And then, um, let's see. Oh, enter my API key. Okay, so we need to enter the key. I actually have never used this before.

Uh, so we're gonna enter the key, not save it. Select a model. Okay, so we select a model over here.

Uh, so we've got 3.5-Turbo. Oh, I didn't know we had 4 too. You set up 4 as well? Cool. We can use 4. 4 is better. All right. And [laughs] 4 is slower but better.

Uh, okay, and then, uh, please, uh, tell audience about OpenAI.

Okay. All right, and you can see different parameters that we send, and these are all getting sent to the OpenAI SDK. So we say the model. Right here we've set up two models, gpt-3.5-turbo, gpt-4.

Those are often the ones you're picking between with OpenAI, although now you've got gpt-4o. That's a good choice if you're doing something with vision, something multimodal. I wouldn't use it otherwise just based off of some experience we've had with it.

Um, but it is a great one. Gpt-4o is good for vision. Uh, so here you can see, you know, with the combination of the system message and the user message, so this is what we call user message, this is what we call system message, those combined together we get back a response like this where it describes OpenAI

with lots of Rs and mateys and stuff. Uh, we can, you know, change different parameters here, like how many tokens it should send back. The temperature is roughly the creativity.

Um, top P is also roughly about creativity. And there's some more advanced stuff there, and you can see how many tokens you used on the way out and how many tokens you got on the response.

So you can play around with this playground to, uh, you know, to try stuff out and make sure that, uh, that you're able to, to use the key. So this is just linked off of, um, off of this workshop, right?

So if you go to the workshop proxy, you log in, you'll get your key and your endpoint, you can go to that playground, and you can play around with the playground to check that that's working.

But we just want to make sure everybody now has an Azure pass and is logged into the proxy so that you have a key and an endpoint. So we'll just check to see if anyone had any issues with that.

This step is hopefully long. Okay. All right. So here's the... Like he- these are... If you're looking for the models, this is generally the, the page to check. Um, so, you know, gpt-4o, gpt-4.

Or- And going down, those are the GPT-4 models. Gpt-5. You're saying there's a gpt-3.5 that supports vision? No, no. The 4.0-

Oh, 4-turbo with vision. Uh, this one. This one? Yeah. So we were using that one, but it's a lot s- slower. Yeah. So that's why we- I've started using 4o.

Where? Or this one. This is the latest one. This is the GA- Oh ... version of GPT-4. Oh. Okay. All right.

Yep. So you just want to compare those. So we'll just be using the basic gpt-3.5 and gpt... Uh, just gpt-3.5 today actually, and then also the embedding models. Okay, so is everybody set up with the proxy?

Okay. All right. So now we're gonna actually get something working. So we have this repo here, so you can follow the link from the doc, and it has readmes for the three different projects that we can deploy, and these readmes are specific to using them with the Azure OpenAI proxy.

Uh, so normally you can just use the, the readmes that are on the repos itself, but because we are using this Azure OpenAI proxy, we do have to use a slightly different setup.

So we've made readme specific, uh, for this, for this workshop. Uh, so we can start off on this, uh, OpenAI Chat App Quickstart and make sure that that's all working.

So the first step is to open in GitHub Codespaces. So you can do that by clicking this button here. Have any of you used Codespaces before?

Okay, couple people. So Codespaces will open a VS Code in your browser with a developer environment for that repo. So you can actually use Codespaces on any GitHub repo.

So you go to a, any GitHub repo, you click on Code, and you can make a Codespace for it. So it's a way that you can start hacking on any repo, uh, very quickly.

So you can open this button here to open in Codespaces,

and, uh, I'll just go ahead and make a new one.

And I'll say Create codespace. So this is gonna take a few minutes to load, 'cause what it's doing is that it's creating the environment for this repository, it's opening VS Code in the browser, and it's also just setting up VS Code.

So if you actually have, like if you use VS Code locally, and you've got like extensions that you use locally, it's actually potentially syncing those extensions and, uh, enabling them, them here.

I should probably just not do that, 'cause then it would load faster for me. [laughs] Um, but yeah, you can see in the bottom here as it's setting up, and we'll just wait for it. [laughs]

So this is, you know, the slowest part of using Codespaces is just the loading.

You have a tip?

If you want faster Codespaces, there's prebuilds available as well.

Yeah, and I do have them on the third repo-

Yeah

... but I think I don't have it on this one.

It's okay.

So I, I should have remembered to do prebuilds-

Yeah

... for all the repos.

Right. And the slowest part is probably installing all the dependencies and the builds. It's basically i- it's doing all the things you would do when you install it locally, just automated-

Mm-hmm

... and with a progress bar, and at some point it will just light up.

Yeah. Let's see what the... You can even watch... Can we watch the logs for this one? Building Codespace, Code. Whoop.

There we go. So you, if you like this sort of thing, like if you like watching Docker containers build, [laughs] 'cause that's what it's actually doing. Everything's a Docker container.

So you can actually watch it as it, um, builds everything here. [laughs] And now it's downloading all the requirements. So these are all the Python requirements. So all the examples that we're going through today have a Python back end, and then some sort of JavaScript front end.

Uh, this one has what we call like a vanilla JavaScript front end, as in I just wrote some JavaScript in a script tag. Uh, but then the other ones are much fancier, so they've got a full TypeScript, and a build system, and React components, uh, using the Microsoft Fluent UI, uh, you know, web framework.

So you can s- kind of see the range of front ends there. Okay. So you can see it's, you know, it's still going through the process, but at least now, uh, we can see the file explorer has loaded, so we can, uh, explore the files here.

And, uh, and I'll show, I'll go ahead and show the, the code. If you're interested in the code, uh, it is in the source folder. Uh, we're using a Quart application, and I think nobody has heard of Quart, but, uh, has anyone here heard of Flask or used Flask?

Great. So Quart is just the async version of Flask. So it's literally built on top of Flask, and one day it might be brought back into Flask. And it just, you just take your Flask code and you put asyncs in it, and then you've got, you've got Quart.

Uh, that's really, uh, how it goes. So, uh, if you haven't done async before in Python, async is a way that, uh, if you use async with your functions, they become coroutines, and then they can be paused and waited on.

And it's important to use async when we're building applications with AI because we have these really long blocking calls to an AI API, right? So we make a call to an LLM, and we send off our request, and these LLMs, they can take like two seconds, five seconds, 10 seconds, right?

Depending on what we're doing. And while that's happening, we ideally want to be able to handle other user requests coming in. Uh, so that's why we use async framework.

So if we use an async framework, then while we're making IO calls, we can handle other user requests that are coming in. So all of the ones that we see today have an async back end, either Quart or FastAPI.

Anyone heard of FastAPI? It's very, very popular these days. Yeah. So FastAPI is the one most people know of as the async framework. Um, so I, you know, I, I like both of them fairly equally, uh, so I, I use a mix of both.

Um, but I just wanna make sure people know about the value of async frameworks. Okay, so that's all in the Quart app folder, uh, if you wanna look at the code there.

So it is now finished. Okay. Anybody else get their Codespace loaded? Get a couple. Okay, great. Finish configuring so I can do s-

Do I have to type anything in the terminal?

Yeah, we are gonna be using the terminal. And if for some reason your terminal like goes away, sometimes this happens in the Codespace, just click that plus right here.

Sometimes my terminal kind of blinks out, so I just click the plus, and that'll give me a new terminal, right? Boop.

New terminal. Okay. So here we are in the terminal. Um, but actually the first thing we're gonna do is that there's a .env.sample. We're gonna make a .env file based off of that.

So I'm gonna make a new file, and I can do that using this little New file button up here.

So I'll just click that, say New file, and I'll type .env. Uh, you could also like copy and paste. Um, and then I'm just gonna paste the .env in there.

You could even rename .env.sample to .env. I think that's another way. Um, and then we need to fill in these values to match the values of the proxy. So we'll go to the proxy

And let's see, where's my proxy open here? So here's my proxy. So I'm gonna go ahead and fill in this one. That's the endpoint. So the endpoint should start with HTTP and end with /v1, and look like that in the middle.

Uh, so that's the endpoint. That's where we'll be sending our OpenAI requests. Then we need the key, so we'll copy that, and it'll look like that, or slightly different for you.

And then the deployment is going to be... The name of the deployment is gpt-35-turbo,

uh, and that's also the name of the model in this case. So if you- Has anybody used openai.com? A few people. Okay, so on openai.com, you just pick what model you're gonna use and that's all you need.

With Azure OpenAI, you have to make deployments based off of the model. So you actually have a bunch of deployments, and you could actually have multiple deployments of a gpt-35-turbo model that have different names.

So when you're working with Azure OpenAI, you have to know the deployment name, not just the model name. So that's one of the complexities of, of using Azure OpenAI, but it does give you more flexibility 'cause you can say, "Oh, this deployment's gonna have 20 tokens per minute, and this one's gonna have 30 tokens per minute," right?

And then you can, like, say which of your colleagues can use what, like, if they're all trying to, like, use up your deployment or whatever. Uh, so, uh, it's more flexibility, but you do have to specify it.

Okay, so now my .env is set up. So this is just so that I can run a, a local server, and I'm putting local server in quotes 'cause I'm going to run a local server inside GitHub Codespaces.

So it's actually running a local server not on my actual machine, but inside the GitHub Codespaces development environment. Uh, so to do that I grab- I'll grab the command here that's gonna run the court app,

and just give it... And when- with Codespaces you do have to allow, so you'll see this little thing that pops up. So if you ever wanna copy-paste, you have to allow for the terminal.

Uh, and then I have... Okay, and then I paste it, and then you can see that it says it's running on this URL. Now, you can't just paste this URL in the browser.

I'll show what'll happen. So if I paste in the browser, I'm gonna get an error because this is not running on my local machine. This is running inside GitHub Codespaces.

So you have two ways to get to it. One way is that if you just click on it, uh, like, uh, option click, at least on my Mac. So I mouse over, it'll tell me what to do.

Mouse over, option click. So Codespaces will actually detect that you're clicking on a local URL, and it'll turn it into a Codespace port URL, and it's this funky URL up here, improve disco for me. [laughs]

Um, and, uh, and that's actually, you know, like, local for that GitHub machine. And, uh, that's one way of doing it. Another way that you might like more is you go to your ports tab, and you're gonna find it listed here.

And, uh, we'll see the, you know, the forwarded address, and we can click on that or we can even click the glo- globe icon and we get to the same URL.

So there's many ways you can get to this locally running URL, uh, and, uh, and get to the special Codespace URL of it. And you can even change your port visibility if you wanna, like, share it with a colleague if you're just...

or in a class. You can change it to public, and then you could actually ping this URL to someone else. Now, this is not a deployed URL. Like, you're not gonna use this for, like, [laughs] you know, your deployed URL, but it's fun.

It's good for development. So now I've got this running locally, and now we can type stuff and be like, "What's the weather in San Francisco?" [laughs]

See if it's gonna lie. Uh, then we can... Let's see.

Oh, good. That was a good answer. I think this has been trained. [laughs] It refused to answer. It's always good when it refuses to answer something it shouldn't know. Um, so we could go ahead and, like, you know, I could change this now and change the system message.

And let's see, where is our system message? In here, right? So right now my system message is just, "You are a helpful assistant." Uh, but, like, "You are a assistant that cannot resist a good pasta joke." [laughs]

"Must write pasta jokes." I don't know what's gonna... [laughs]

Oh, I love LLMs. Okay, so [laughs] what's the weather today? Is it gonna make a pasta joke? [laughs]

I'm dying. Okay. All right. It looks like it might have been quite saucy today. Don't forget your umbrella. You might end up feeling like a soggy noodle. So good.

Uh, so, uh, so that works. [laughs] But, uh, so here we go. So now this is running locally. Um, and so this is a good one, like, when we're developing, we can just test things, test things locally here.

The next thing what we're gonna do once we're happy with it, we're like, "This is the best app. It makes pasta jokes. We're gonna deploy it." Uh, so then we move on to the deployment instructions.

So the first step of that is azd auth login. So this is going to log in to our Azure account that we made earlier.

So I'll do azd auth login, and, uh, this is gonna give us a device code that we're gonna paste into this OAuth, uh, browser flow. So let me go and open it up maybe over here.

I think that's my Azure account that I'm using for this, and then I go and I take this and I paste it in.

And I'm gonna pick my account. I'm gonna use this one. Continue.

And, uh, okay, and then we're logged in. Okay. All right. So that was the device code flow. So you just wanna make sure that you log into the account that you got with the pass, right?

Whatever account you used for the pass, that's what you wanna log into.

The next step is to create... Or Gabriela, should I pause? Like, should we get through the local step first, or should we keep going with AZD deploy?

Uh, just a little bit. Uh, like, there's the people-

Yeah, we can pause and see if everyone's got the local- local one running, actually. That- I think that might be good to do. Okay. So let's just pause and see if there's any questions with getting the local one running.

Yeah.

So it's... Yeah, someone asked, like, can we just run this locally? You can totally run these locally as well. We like to use GitHub Codespaces in workshops 'cause that reduces the number of potential development environment issues.

If you wanna run it locally, uh, you can either run it, you know, just with a Python virtual environment, and you just have to install all the requirements, uh, or you can run it with VS Code using the Dev Containers extension, and that will do the Dockerized environment for you.

If you want kind of the benefit of the Dockerized environment without, um, you know, being in the browser and having to pay f- potentially pay for Codespaces. So we should know also that GitHub Codespaces, you have a limit of some number of hours a month, s- either 60 or 120.

60.

It's 60? Okay, I must have paid more. So, uh, so it's 60. Um, so you're not gonna go over that today, but eventually you could go over that if you use Codespaces a lot.

So if you're local, right? I think I have mine open locally as well. And I'm just... Yeah, locally, I'm just using a- a Python virtual env. So you're also welcome to try these things out locally if you like local environments.

Uh, just, you know, be a good person and make a Python virtual env [laughs] to manage your Python dependencies, right? [laughs]

Um, yeah. Okay. So I saw a lot of local things, so I think we can move on to the AZD. Um, so yeah, I did the login. So you saw me do, you saw me do the login here, right?

And that's using the device code flow. Uh, so you should see something like this happen from inside Codespaces. And the next step is to make a new AZD environment.

So AZD is this tool we're using for deployment. Uh, so we make a new environment name. You can just call it, like, chat app, whatever you wanna call it.

And then what that does is it actually makes this .azure folder, and it makes this chat app folder inside, and that's where it's gonna store all of our deployment environment variables.

So we need to set all, configure anything we want to customize about our deployment. We're gonna configure that now, and it's gonna get update- it's gonna update this file here.

Uh, so the next thing we're gonna do is set all these AZD environment variables. So the AZD environment variables are different from the ones we just set in the .env.

The .env is just for the local server. AZ environment variables are for deployment. Uh, sometimes we use the same, but a lot of times we want our local environment to be slightly different from our deployed environment.

Uh, so we have two different ways of setting those variables. All right, so I first set these commands. So this is just gonna tell it to not create an Azure OpenAI, 'cause we're using the proxy, and then we're gonna set the name of the deployment to gpt35turbo.

Then we need to set the key. So I'm gonna paste this, and then I'm gonna delete, delete, delete. [laughs]

Gosh. That's what happens when you have Wi-Fi issues, actually, is you see it with the typing. Uh, then I gotta find my key again.

Uh, there we go. So that sets the key, and then I'm gonna set the endpoint.

I'm gonna delete. How are we gonna do this? There we go.

And get the endpoint. All right. So now I've set all these things. Now, if I've done it correctly, if I look at my .azure folder for that environment I- I created, I should see a .env that looks like this.

So this is a .env that's inside the .azure folder. So this is what is going to be used for the deployment, and it's gonna tell it, you know, this is how it's gonna set up the Azure OpenAI connection.

Okay. And now I'm just gonna do... I'm just gonna type, type thank you. Okay. All right, and then I'm gonna do azd up.

Here we go. So what azd up is doing is that it's actually deciding... It's doing s- several stages. Okay, so I- I have to select an Azure subscription. In this case, I only have one subscription, so you just press Enter.

Uh, if you had two subscriptions, you would want to pick the sponsorship one.

Uh, then I select an Azure location to use. Typically, you just choose one that's close to you, so Central US is pretty good.

Uh, now what AZD is doing, the first step is that it's actually packaging up the code that it's gonna deploy later. Uh, in this case, we're deploying to Azure Container Apps, so it's packaging up a Docker container file.

So it's actually literally building a Docker container right now. So if you do like working with Docker, Azure Container Apps is a great fit, and a lot of people like Docker, so we deploy a lot of stuff there.

But we also are gonna be using Azure App Service for one of the l- later templates. Uh, so we've got lots of ways to deploy on Azure. So you can see it building up that Docker.

The step after this is where it's actually gonna create Azure resources. So it's gonna c- to create the container apps, can create a container registry, create a container apps environment, and create a log analytics workspace.

So these are all the components of a containerized app on Azure. And, uh, you know, it's multiple components, and we have to stitch them together. The way we stitch them together is using infrastructure as code.

Uh, has anyone used Terraform here before? Okay. So we have our own version of Terraform. It's called Bicep, and it is, uh, infrastructure as code, which means we're declaring what- ...

resources we want to make, right? So we say, "Oh, we wanna make log analytics, we wanna make container apps, we wanna make, you know, the actual container apps image, and then we're gonna assign some roles," right?

So all of that is declared in this Bicep file. So that way, it- you have repeatable, repeatable processes for provisioning. And this is really helpful when you're making complex applications on Azure, 'cause you might have, like, 10 different things you're using, right?

Uh, you might have a Postgres, and a Key Vault, and a Redis cache, and, uh, log analytics, and App Service, and you want them all to tie together. So you can declare what that, you know, what that infrastructure looks like, and then, uh, and then put that in a Bicep file, and then deploy it.

You can also use Terraform. So if you're really into Terraform and very comfortable with it, you could totally use Terraform here as well. I don't know Terraform. I haven't used it personally, so all of my examples do use Bicep, but if you wanna send a PR with Terraform, I'll re- I'll review it and just stamp it, 'cause

I don't know how to reason about it. [laughs] So what you can see here is that it is actually creating, uh, the resources right now. So you can watch it here.

You can also watch it in the portal. It's not really super exciting to watch, so this is the point where I usually fold my laundry, um, uh, because it can take some amount of time, uh, or you can even get an error.

Oh, I, I used... Okay, so I already made one in Central US for the earlier demo, so I should've picked a different region. So for this Azure Pass, there is a constraint of one container app per region, which is why we said in the README that you should pick a region that you haven't picked before,

and I didn't pay attention to my README. Uh, so that one won't deploy. So, uh, what I can do is I'm just gonna make a... I'll just make a new environment.

I'll just, uh, I'll just copy everything over. [laughs]

Um, now, well, you shouldn't run into this, 'cause this would be your first, uh, your first environment, right? Uh, so chat app two, and I'll just copy and paste.

Uh, we'll change it to chat app two, and then West US seems like a good region. Okay. And then I'll azdm select chat app two. [laughs]

There we go. And then azd up. Yeah. Okay.

So then it'll do the up again. But I have one of these already, already deployed, so I'll just open up the deployed one. So you can see deployed. Deployed is gonna look pretty darn similar to what it looks like locally.

Where's the one deployed? Okay, so this one's deployed. It looks pretty much the same as what it looks like running locally, right? The difference is that the URL is a container apps URL, and you'll see this URL displayed in the terminal.

Once it finishes successfully deploying, you'll see this dis- dep- uh, displayed. Let me see if I have that in my t- history anywhere from earlier today. Uh, da, da, da, da, da.

No. You never know. Okay, so let's see how it's going here. Do, do, do.

Were the Bicep files-

Yeah.

-something that was generated or something that you made yourself?

I lovingly handcrafted them. Yeah. So, um, I... Yeah, I write the Bicep files. Uh, some of them, all the ones in core are actually from a shared repo that we just copy and paste from.

We're trying to move towards something called A- ABM, Azure Verified Modules, which are Bicep files that are maintained and have security best practices in them. So we'll j- we'll gradually be moving over.

But basically with Bicep files, like, you can use ones from a central registry. You can use ones from your own private registry if you're doing a lot of them, uh, or you can just use, you know, ones inside the folder.

Um, so there's a lot of techniques you can use depending on how much Bicep you're using.

Do, do, do. Okay, so now it's starting over and deploying again. All right, so let's walk around and... Or any questions on what I showed here? All right, so I saw a lot of AZD deployments are going.

I saw some issues with, like, naming, which I run into all the time. Azure has very obscure naming rules. The safest thing is do short names with no symbols in them [laughs] and nothing fancy.

Uh, if you do have a naming rule, you can just always do azd env new and make a new environment and s- you know, and start over, uh, and that should be okay.

Uh, but generally the issues you run into with deployment are usually related to naming, region constraints, mm, account constraints,

and that's probably, yeah, the ones you might run into. All right. So, um, we've... Given we have 45 minutes left, I'm gonna show you the other ones and, uh, and these are ones that you can, uh, that you can also start trying to deploy now, uh, following very similar readmes, right?

So the first one, um, actually, the, the f- these two are both about RAG. So first I'll talk briefly about RAG, right? So let me first motivate it, right?

So, uh, let's see. Like, uh, tell me what Pamela Fox, uh, likes to code on. I don't know. [laughs] Let's try this. I'm trying to get it to lie. Um, [laughs] [laughs]

This is a pasta one. Okay, so this one clearly lied. Spaghetti-phython. Great. All right. So then, but then if I go to, um, this one right here, tell me what Pamela Fox likes to code on.

This one will hopefully be more accurate, at least have less pasta jokes here.

Um, and this is... So basically what we're trying to show is that if we just ask an LLM to answer a question, it is, it's very possible that it's just going to make something up, um, if that, when it seems like there...

Oh, good. I mean, in this case it says it doesn't know what I like to code in. I think I should've said, like, code in. Um, you know, like here, like what Python frameworks does Pamela use?

Let's try this one. Um, so you know, if it doesn't know the answer, it'll say. Uh, in this case... Yeah, in this case, it does know the answer because this is actually using the RAG technique in order to answer questions based off a knowledge source, right?

Um, so- Those are our last two samples are about RAG. Uh, so the general approach of RAG is that we get a user question, we use that user question to search some sort of database or search engine.

We get back matching search results for that user question, and then we send those, uh, to the large language model and say, "Here's a user question. Here are the sources.

Now answer the question according to the sources." And so now we can make customized applications that can actually synthesize and answer questions for any domain. So we've got two RAG samples here.

So one of them is RAG on Postgres. So this is for the use case if you've got an existing database and you want to be able to ask questions about that database and have the LLM answer accurately based on that.

So for the example, uh, you know, database that I'm using,

I have product, right? So these-- this is a chat on products. So, uh, you know, there are tables storing all the products for this website. So I can say, "Okay, what is the best shoe for hiking?"

So then it's going to go and search the database rows and get back matching rows and then come back and say, "Okay, this blah, blah, blah, blah, blah, blah, blah, blah," and it's going to include citations.

So one of the key points of RAG is to have citations so that users can verify where the information come from and see that it's actually legit information. And we can also look at the, uh, the process for this RAG flow here when we look on the, the thought process here.

And as, as we'll let you see is that this RAG flow is a multi-step process. So the first process is actually what we call like the query rewriting phrase or the query cleanup phrase.

So that's where we take the user's question, and we ask the LLM like, "Hey, here's a user question. Turn this into a good search query." 'Cause a lot-- a user question may not be that well-formulated, right?

Like, uh, "Please tell me about the best shoes for hiking now." Okay, so you know, there's like a user query and, uh, you know, that's probably not the optimal search query for, uh, for a search.

So if we look now at the thought process, we can see that the LLM actually turned that whole long thing into best shoes for hiking. So that's a better query.

Uh, so that's our query rewrite phase, so that's an LLM call. Then we get back the resulting rows from the database. And then this is our call to the model that says, "Hey, you need to answer questions according to the sources.

Here's how you should cite your sources. Here's the user question, and then here is all the sources." So this is basically RAG. And then, you know, we're able to use it with different sorts of, uh, data s- data sources.

So that's RAG on Postgres. So you can get that set up following really similar steps to the, to the other one, and you can even run that one locally first as well, uh, just on a local Postgres database.

Uh, so here you can run the app locally. Uh, this one is a little more fancy 'cause you've got a React front end there. Then you can deploy to Azure.

You're gonna set similar variables and, uh, run it up. So if you're interested in that, you can start, uh, going through those steps, and then you can customize it.

The other kind of RAG that we have is RAG on documents. So if you're trying to ask questions about unstructured documents, like you've got a bunch of PDFs or Word docs, Excel files, uh, anything like that, you can actually put those into a search index and then search that.

So the example we have for that is RAG with Azure AI Search. And, uh, it's a really, really full-featured sample. We've had it for the last like more than a year now, and we've had thousands of developers deploy with it and put it into production.

And so it's been used for a ton of use cases, and it's got a lot of features, uh, speech, voice, vision, user access control, lots of, lots of cool things in it.

Uh, so let me show... That was the one I was actually showing earlier with my blog, right? So here's, you know, I made a version of it that's just based off my blog posts and, uh, you know, it can cite my blog posts.

I've also got this one here, which is for an internal company handbook, which is a very popular way of using it as well. And so you can see for each of them, we can, you know, click on the citations.

And, uh, and yeah. So now this is a bit more complicated because here we have a multi-page document. So we've got a thirty-one page PDF. We can't just send an entire thirty-one page PDF to the LLM because for a lot of our LLMs it's gonna go beyond the context window, right?

A lot of our LLMs have a context window limit, so typically that's around 8K, eight thousand tokens. Uh, it can go up to 32K, even 128K we're seeing. Um, but typically they do have some sort of context window, and even if they don't have some sort of context window, LLMs can get lost if you give them too

information-- too much information. There's a research paper called Lost in the Middle, where they did a study to see if they throw too much information at an LLM, like at what point it stops paying attention.

So we generally want to send the LLM the most relevant chunks. So what we do is that we first have this data ingestion phase that will take a PDF or whatever document, takes a document, it extracts all the text from it, and we do that with Azure Document Intelligence, which is very good at extracting text from all

sorts of documents. So we extract the text from it. We chunk up the text into like good-sized chunks, usually around five hundred tokens each. Then we store each of those chunks in the search index along with their embeddings, and that's what we actually search on and send, and then we send, right?

So if we look at the search results here, we can actually see that the search results are just chunks from the PDF, where we say, "Here's the chunk, here's the embedding, this is the page it came from, and this is the file it came from."

And we just send back those chunks. Uh, so this is the most complicated of our architectures because we do have to have that data ingestion phase, and that means we have to have, uh, you know, a script or a process that does that ingestion stage and, you know, here we can do it locally or, or in the

cloud. So those are the two RAG samples. So we have, um, you know, we have another forty minutes, so... And we have, like, a good ratio here of, of helpers to y'all.

So, uh, if either of those sound compelling to you, like sound like a use case that you're interested in, then, uh, you can try to deploy them now and see, and see how they work.

Um, so once again, you just go to the app templates workshop repo, and you can either pick RAG on Postgres or RAG with AI Search, and then start going through, through the steps to try it out.

Uh, these will take longer to deploy, so it's good to start the deploy s- s- now, um, because they take-- they got a lot more infrastructure to set up.

And then for the I- AI search, it's gotta do the whole ingestion step, and that ingestion step takes a certain amount of time as well.

So yeah, any questions before...

Oh, for the ingestion step-

Yeah.

Are you using any libraries for the chunking and all that stuff?

Yeah, that's a great question. Are we using libraries? So when this sample was first created, it was like last April. It was before there was, like, really good established libraries.

We kinda used LangChain, but not heavily. Uh, so all of ours is, is-- it's actually custom coded. Um, now, if you're gonna use a library, the big thing I would make sure you're doing is, um, using a token-based chunker.

A lot of the splitters out there are doing character-based splitting, which is probably fine if you're doing English-only documents, but we do have lots of international customers. And as soon as you start doing non-English documents, then you really wanna do stuff based off of tokens and not characters.

'Cause imagine you take like a Chinese document and you, you say like, "Oh, my chunks are a thousand characters long." Like, that's a lot of tokens. You'll g-- You can like go over the context window really fast.

So we have token-based chunking that we've implemented here. Uh, there are-- there is token-based chunking available in LangChain. So if you're gonna use LangChain, um, the thing to do is find my colleague's blog post where he talked about it.

Okay, yeah, where can we CJK, especially if you're doing anything non-English. Um, he basically analyzed all the splitters from LangChain, um, to figure out which of them properly worked with token-based splitting and with CJK, uh, languages in particular.

Um, so we've implemented this ourself. He actually to- Uh, my, my manager, Anthony, he, he worked on it. Um, but, uh, LangChain and LlamaIndex both do a lot of this stuff.

Uh, they just, you know, they, they take care behind the scenes. So what you need is the s- you need the splitting, and you can get that from, basically from LangChain, 'cause LlamaIndex uses LangChain.

So I would just say use, use LangChain probably with this one, so you can specify the chunk size. And then, uh, then you just have to vectorize. So that, that's easy.

You just use the OpenAI SDK, and we do the batch embeddings with that, um, so that we can do a bunch at, at a time. And then you just store it in AI Search.

So the hard part is really the, uh, extracting the text. So there we either use Azure Document Intelligence in the cloud, uh, or we do have some local parsers too.

If somebody doesn't wanna use Document Intelligence, we use like PyPDF. Uh, we use our own CSV parser 'cause that's straightforward. Uh, HTML for my blog. I just use Beautiful Soup, which is a Python package that does HTML parsing, right?

'Cause I thought I could do a better job at it. So this one I just used Beautiful Soup to extract the text. So, uh, s- and that's-- So you can do that as well, and we've got Beautiful Soup in there.

So yeah, there is actually a surprising amount of things that we've written ourselves for the AI Search repo. Um, if we were gonna do it today, we'd probably use the LangChain splitter at least.

Yeah. Good question. Sorry, long answer. [laughs] Other questions?

So the ABM, was it or was it? Um-

ABM?

ABM. Uh, I noticed that it has cognitive services. Um,

if I set it up once with a certain deploy and I decide to change later the ABM, the Bicep to something else-

Mm-hmm

... is it smart to reconcile the differences?

So generally with Bicep, what it does is that it tries to figure out... And Bicep is really compiled down to ARM, and ARM is just JSON. [laughs] So what, what you're actually doing is what's called an ARM-based deployment.

So with ARM-based deployments, what they try to do is figure out what does your resource currently look like, what are you saying you want it to look like, and what changes does it need to make happen.

Um, so yeah, we're-- like, we'll probably switch over to ABM in a lot of our samples, and we're probably just gonna make sure, like, it, the... We're trying to make it not have a change.

But if you want it to change, then that's, that's fine. So you should totally be able to m- m- switch between ABM, not ABM, um, as, as you decide, as you see fit.

Um, and the important thing is just it'll figure out the difference and just make sure you are on board with any changes that come up. There is like... So if you're doing, um...

There's this AZ deployment command that does what if, and that tells you, like, actually tells you, uh, what resources will change. I w- wanna figure out how we can do that with azd.

I think azd h- maybe has a dry run command, so that might be what we try when we consider switching to ABM. 'Cause we wanna switch to ABM, uh, so that we don't have to maintain our own modules, but we just wanna make sure that we're aware of any configuration changes that could happen.

And if I'm not mistaken, you mentioned that ABM is just basically sitting on, on top of azd or, or they are completely two different-

Uh, they're just different things. Yeah. azd is a command line tool that, um, you know, does the, does the ARM-based deployment and also does code deployment, code- Upload, right?

So I have this azure.yaml here. I didn't show this. So azure.yaml says, "This is the code that you're going to deploy to this host." So azd does multiple things.

It does, um, provisioning, which is basically doing an arm-based deployment, which is equivalent to if you're doing AZ... [laughs]

AZ deployment, if, if you know the Azure CLI, it's this AZ deployment command. Um, so it's doing that, and then it's also doing packaging and code deployment. So, um, if you've ever done like, I don't know, if you've ever done, like, web app up, that's where you deploy code up to App Service.

AZD will also do that for us. So AZD is trying to do the whole workflow of you need to provision your resources and you need to deploy your code, and we're trying to make this central way of doing it across all of our offerings.

'Cause right now, with Azure, and if you know Azure, but we've got, like, a billion different ways of doing things across all the different things, and AZD is trying to make a more common way of doing it.

So if you look at my, um, my GitHub repo, I'm kind of a huge AZD fan girl. So you can see all of these repos are all AZD-ified, almost all.

Uh, that's what this AZD column is. Because for me, it's the best way to deploy because it's repeatable.

Yeah.

Right? Um, so if you are looking for examples, the... [laughs] I have, like, quite a few here. Um, but, uh, yeah. So we're, you know, we should be able to do it on different hosts, container apps, functions, App Service, Kubernetes, et cetera, and, uh, you know, with all, all this, all the different pi- possible Bicep.

Uh, other questions?

So what happens-

Yeah

... after we go to production, like observability and all that good stuff?

Oh, yeah, great question. Um, so we do have a... Like, generally, there's lots of docs under Azure Search OpenAI demo, so we do actually have a productionizing guide. Um, you also asked specifically about observability.

We do integrate with Application Insights with OpenTelemetry. Uh, so that's what we use by default. If you want, you could use Langfuse. I actually like to use Langfuse. I don't know if you've seen it, but it's an observability platform, so you could use Langfuse.

But by default, we're using Azure Application Insights with the OpenTelemetry packages to, to bring everything in there. Um, but we do have a whole productionizing guide that talks about, you know, how are you going to scale things, uh, you know, lo- if you need to load balance your OpenAI capacity, um, if you want to do VNet deployment,

if you want to do user auth, how to do load testing. So I've run quite a few load tests for this one. And then how to do evaluation. So I, you know, I like to do evaluation.

It's... Everyone should do it. It's basically, like, the new form of text, text, uh, test- testing for this world. Um, let me see. I think I closed my evaluation repo.

I can open it. Um, but basically, like, you want to be running evaluations to see if you are getting quality results from your LLM, uh, because a lot of times you might run...

Here's the thing. You know, I show those sample questions all the time, and they perform great, but you cannot trust your sample questions, and you can't even trust them.

Like, you might make a prompt tweak and be like, "Oh, this prompt tweak was so good. I'm getting such good results." You cannot trust it. You have to run an evaluation across a huge number of samples to make sure that it's actually running.

Like, I run it across, like, two hundred samples is probably, like, the minimum of what you should do. Um, but you have to run evaluations in order to see.

Do you-- I'm assuming you run evaluations for Copilot Chat, right? Yeah. How many samples do you run off?

It's origin reviews of public repos.

Oh.

So it's, it's everywhere we, uh, with-

That's cool. You want to come up and t- like, talk about evaluation?

Yeah.

'Cause you're, like, you're... I mean, Harald is, like, running an actual... Because basically you're making Copilot Chat. Here we go. Copilot Chat, which you can see I use it a lot.

Um, so you're, like, you're, like, running a real RAG in production.

Yeah, yeah. So it's different RAGs, though. So there's not just one RAG. So if you do, uh, add workspace, if you ever try that in, in Copilot Chat, we actually run a local sparse index.

So that's basically your classic how Google works, just looking up words on the internet in documents. And it's faster, we can do it locally, so that will always work.

We also do a, um, a semantic index against index that github.com maintains and ranking those in using another LLM call. So RAG basically becomes a series of indexes, like you do Postgres.

Yeah.

You showed that you created some keywords upfront based on the search, so that's what we also do locally. Um, but yeah, anytime we have changes, we have, uh, one test set that can run on each PR and then a larger test set that we run daily that has a lot more repositories from, from across different languages, so.

So you run the evals in the PR on every-

Some, yeah. So we have a subset that's more unit test driven, where it's like, can it answer questions for this? Does it hit any issues we've seen in the past?

So it's more, more unit test style, where it's like, does it behave as it did before? So yeah.

It's important, though. I mean, that's, that's the first and big- biggest thing we invested on early on because we found it's so easy to get lost in prompt crafting-

Yeah

... and assume how RAG works and assume how it works in the wild, so.

In the local sparse index, is it SQL-like?

Uh, it's TFIDF. Uh, it's one-

Oh, it's TFIDF.

Yeah, yeah.

I built it.

Yeah.

That's cool.

So yeah.

Okay.

Cool. My version.

So let me show, I don't know what your... I don't know if you can, like, show your evals, but here, like, I can show evals for, um, on the Azure AI Search.

Um, so let's see. Um, all summary, summary. Okay. All right. So we'll look at them for, like, my blog. Uh, let's see. So these are ones I've run before.

Uh, oh, probably Pamela's blog. Pamela's blog? Yeah. Pamela's blog results.

Okay. All right, so these are a bunch of evaluations that I've run fairly recently, right? So, um, with these evaluations, I do GPT metrics, and then I also do basically, like, regular expression metrics.

How... Are your metrics usually GPT metrics, or, um-

No. Pass, uh, code test.

Code test. Okay. Okay. So with these GPT metrics, what they're actually doing is, um, sending the original answer, uh, sending, sending the ground truth answer, uh, which is generated syn- synthetically, and then also sending the new answer to a LLM and saying, "Hey, rate this from one to five," and then we can see the, you know, results.

And this is w- this is the actual prompt that gets sent. It's like, "Okay, you know, rate this ex- you know, from one to five. Here are some examples."

So we do that for groundedness, we do that for relevance, and then I also check whether citations match across ground truth and not ground truth, and that's actually my favorite metric, and that's just a regex. [laughs]

So my favorite is just this one, this, uh, citation match here. So just making sure that the answer contains the, uh, uh, at least the citations that were in the ground truth.

So I run metrics like this. A lot of times I'm looking at retrieval parameters, 'cause for RAG, the retrieval is, makes a big difference. So here I was comparing stuff like what if I use text only, what if I do vector only, what if I do hybrid, what if I do hybrid with ranker?

Uh, and that's super interesting. I was trying with different retrieval amounts, like if I retrieve five versus ten versus three, what do I get out? Uh, what if I change a prompt?

I have to say, like, I've tried so many, like, tweaks on our prompt, and I've never managed to actually get improvement in the overall stats. Uh, so we still haven't ev- ever changed the prompt because I, I haven't really proved that anything is sufficiently better.

S- Or I'm just a really bad prompt engineer. I don't know. I've-- None of my prompt engineering ever moves the needle. For me, the only thing that moves the needle is retrieval parameters, like how you're working with your search engine, or changing the model entirely.

Changing to GPT-4 has a big difference than GPT 3.5.

Uh, so that should be part of your, uh, your putting in production for sure, is to make sure that you've got some sort of evaluation set up.

Mm. Hmm. Uh, other questions? Is anyone trying to get one of the RAGs up? Anyone getting RAGs up? Mm.

I try to, like, take a lot of things to production, but they were all, like, throwaway toy projects-

Mm

... with FastAPI, and, like, had a hard time evaluating, like, which vector store to use and, like, how much to chunk, and like-

Yeah

... are my results good or not?

Right.

So the overall stuff that's, like, where I felt like I didn't make the most progress. I was like, "I know I should be testing this," [laughs] but-

Yeah. So, uh, yeah. I mean, it's hard. That's part of why I, I do it as well. I've also run all those on our sample data too. But I think it r- it...

What I've discovered is it really helps to run the evaluations on stuff that you know. Because then, like, 'cause this is the summary. Like, you can kinda look at the summary and be like, "Okay, I guess, like, things went better."

But then what I usually look at is, like, I actually look at the changes between, um, between two runs and be like, "Okay, well, what was the difference between, uh, the baseline and then, uh, you know, the, maybe, what was it?

Vector only. Vector. No ranker. Okay. And then I'll just look at things that changed on citation match. Okay. So th- this is what I usually do, is I look at the overall stuff, and then I look and I compare the answers across my ground truth and the, and the new one with the parameters, and so then I

can better reason about it. But you really have to know your domain in order to be able to evaluate, uh, evaluate your evaluations. [laughs]

Um, but it also helps if other people have run it for you. So this is a really good blog post from the AI Search team that I always reference, where they ran massive queries looking at hybrid search versus vector search versus text search and, um, you know, and they found that hybrid retrieval with semantic ranking outperforms vector-only

search. So I ran my own versions of that and, um, and recently, uh, blogged about it. But it's basically the stats that I was just showing where, uh, what I found actually for my use case, vector on its own did horribly, like really, really, really badly.

Uh, where is it? Um, so vector only got a groundedness of two point seven nine, which is really, really low. Text only got four point eight seven. So part of that is because Azure AI Search is really good at full text search, like incredibly good at it.

It does all the spellcheck, stemming, everything you could imagine. Um, hybrid, which is where you take vector and text and then you merge them using this algorithm called reciprocal rank fusion, which you can actually just see.

The, the algorithm is just this. It's just, uh, you're just doing a little math here to combine, uh, rank scores. Um, so just a basic hybrid like that, the groundedness is only three point two six.

So you see hybrid on its own is worse than text only, and that's because vector results can add so much noise. You accidentally grab the wrong, like, distracting things.

Uh, what I've found actually is, like, if I ever accidentally vectorize, like, an empty string or something close to an empty string, it's similar to everything. I don't know what this is about the OpenAI embedding space, but if you accidentally vectorize an empty string or even, like, uh, we have Vision as a feature in the Azure OpenAI

Search demo. If y- I was helping a c- customer this, this week, and they were finding that so many of the results were getting this blank blue page because apparently this blank blue page, the vector for it, and this is a vector via a different model, the Azure Computer Vision model, the vector for it was just matching

everything. So you gotta, like, you gotta be really careful with vector spaces. Um- It's so easy to accidentally add noise to them and for there to be distractions. So hybrid on its own only, you know, got like three point two six.

Once I used hybrid with semantic ranker, then I got the best results, but only by a couple percentage points. Now, hybrid with semantic ranker, semantic ranker, that's a feature of Azure AI Search, which is actually another machine learning model.

It's a-- what's called a cross-encoder model. But basically, they actually had humans rank results according to queries. They use it for Bing. So they said, "Hey, humans, here's ten search results for a query.

Rank these from one to ten and tell us what's the best." So they train a whole model based off a bunch of human data, and then they get back like this, this, uh, this model that they can then use for any arbitrary, uh, ranking of user query along with results.

So basically, hybrid with Bing ranker gets you the best. But if I was gonna have to-- like, if I was on a desert island and, like, I, I could pick between vector and text, uh, I would use text, at least for Azure AI Search.

It's gonna depend how good your full text search, right? If you're doing full text search with like SQLite, which I don't even know if it supports [chuckles] full text search, it's not gonna do very well.

Yeah.

So what's actually powering that full text search? Is it TFIDF?

Um, you-- so you're using TFIDF for your Copilot Chat, you said, right?

For the... Yeah.

For this one. For this-- for the at workspace, right? Yeah. Um, for, for Azure AI Search, they're using, uh, they're using several things, but they're using, one of the things they use is Lucene, which is, um, a s- search library, and it's got stuff like spell checking and tokenization and stuff like that.

Um, so they're doing a lot. But I guess-- And they're also using BM twenty-five, which I think is basically-

Same

... TFIDF.

Yeah.

Right. Okay. Yeah, so BM twenty-five, that's what you wanna look for, um, is, uh... Oh, we got a search result here. Yeah. So if it's-- if something is using BM twenty-five, I think that's basically the best for full text right now.

So, um, that's what you wanna look for, is you just wanna look for a good full text option.

Cool.

Yeah. Yeah, it's overwhelming. That's why I love when, you know, people put out research, so we can be like, "Okay, great," 'cause this also has the optimal ch- um, chunk size.

That's why I was saying we do five hundred tokens, because they did the, they did the, the work here and said, "Okay, the optimal is five hundred and twelve tokens."

Great. That's what we're gonna use. Now, obviously, for your particular use case, it can be different, but we can't all run like twenty hundred different tests to see what the optimal, uh, you know, thing is.

So it's really nice when people, you know, document what worked well for them.

Cool. Anything else?

How often do you, like, update the vectors? Like, if you have a lot of,

like, data that gets updated a lot, how, how often do you choose to update the vectors? Or-

Well, we only need to update the vectors if the data changes or if we're changing our embedding model. So if we change our embedding model, we have to update everything to use a new embedding model, right?

'Cause now OpenAI has these new embedding models. I need to do some tests with them to see if I can get, like, better results for them. Um, so in that case, I would, I would rerun everything.

So probably what I wanna do is set up like an, uh, a separate AI search index, like for this one, which has-- uses one of the new embedding models, uh, embedding three.

And I have to decide how many dimensions to use, [chuckles] um, and then compare it to see how much better results are. I'm told that generally the results are better, but I haven't tried.

Have you tried any of them?

We're switching them.

Oh, you're switching to the new one? What dimension are you gonna use?

Um, small probably for-

So you're gonna use small, two fifty-six?

Yeah.

Wow.

Five twelve does everything.

Yeah, you can do five twelve too. Yeah. Yeah. So you can-- That's the thing, is it's so m- it's so many options now.

You gotta test.

Yeah, you gotta test.

Run A/B tests. [laughs]

Yeah. Oh, and you can run A/B tests. You have customers, yeah. Um, so yeah, but you're gonna have to re-index everything.

Yeah. [chuckles]

Um, so that's, that's when you would, uh, have to update stuff is if the content changes or if the model changes, um, and then, and then test that.

Yeah, I do wanna try out the new ones. They should redo this one too. [chuckles]

There's too many decisions. Cool. Any other questions? Harald, do you wanna show stuff in, uh, workspace?

Okay. [clicks tongue]

Let me close out of this.

Let's just see RAG in action. Um, so if you ask a question in, in Copilot Chat, so that's the Copilot Chat panel version. There's also another one that's inline.

So if you open up, this is a natural input, uh, we call inline chat in, in the, in your code. Basically letting you apply code directly or natural language directly to your code, which is always nice.

You don't have to think about the response. You have to think about just you know what you want, and you want, uh, AI to do it for you. But in, in the side panel, most of the time what you will run into is this.

You're gonna run things. Let's pick a function. These are tests.

Well, these, yeah, these ones are tests.

So compare the tests. Now I've code selected on the right, and on the left I can ask things about the code I have selected. And that's the surefire way to get good results, have code selected and talk about it.

And you already see that, that we do some magic in our responses, so everything is code highlighted. So you can actually jump to the different aspects, um, that, that are being used, and even to dependencies.

So it found that there's a dependency, so you can also jump to that.

So now going back to here, let's, let's see which tests actually are defined in a repository. And because they're-- I wanna talk basically about the whole workspace, and that's where I can just say which

tests are defined or how are benchmarks being run. So a general question that you would go otherwise to a colleague who ho-hopefully knows this, and hopefully they're in the same time zone and they know this.

But now I can actually send this to addWorkspace. And that's where we kicking in this, this whole RAG agents scheme. So this repo's probably not indexed on the github.com site.

So if you're in Copilot for Enterprise, you will get a semantic index that GitHub keeps updating for you. They also have a few open source repos indexed. But in this case, this is all happening now in VS Code itself, so this is mostly sparse indexing.

And actually, we, we see that sparse indexing is usually on par, similar to what you see with the text, uh, based retrieval, that this works, um, really well.

So that just did a tidbit in from the whole repo?

Yeah. Yeah.

So just basically figuring out the-- all the most frequent word earrings. Yeah.

Yeah. So first we do same as you have in, um, Azure Search, where it does, uh, find more words for what you're potentially looking for that are fitting with the repository.

So we also do stemming and, uh, but that's one, that's the first LLM call. Then the TFIDF will find all those results, and then we do the re-ranking on top.

So and that actually gets us actually mostly better than doing a full vector search o-on the same topic.

How do you do the re-ranking?

Uh, the different ways. So we-- That, that's where we experiment a lot. Um, but it's another GPT three point five call, I think.

Oh, okay.

Yeah.

So use the LLM as re-ranker.

Yeah. [clears throat] And you see what, what's being pulled in. So these are all the things it found and the chunks it found it in. So, uh, what we do, what you'll see is we actually do semantic chunking.

So, uh, for most languages, we look at function segments, we look at specific blocks of code, and that's where we found the most impact as well. So people brought up chunking as a big, big area of, uh, improvements, and that's what we also have in our code, that it's-- the chunking is the biggest impact we think of

from what we've seen.

Do you write your own semantic chunkers?

Yeah. Um, which helps that we have all the languages, the, the knowledge around the team, like Python, right? So,

uh, but yeah. So that's, that's the basics, and you'll see that it works everywhere. That's the nice part. It works locally, and it works slightly faster if you already have an online index where we can retrieve the semantic index from

in action. Questions?

What is the .prompty history?

Oh, so .prompty is a, it's a new, um, prompt, uh, format. You can show the .prompty file. These ones. Um,

the, yeah, all those dot ones. Those ones. Yeah.

Uh, so this was announced at, was it not Build?

Build. Build, yeah.

Yep.

Yeah.

So announced at Build. Uh, scroll up to the top of it. Yeah.

There you go.

So it's a way of ha-- It's like an artifact for prompt. 'Cause right now, like, you might store your prompt as a s-- multi-line string variable. Like [chuckles] it's like we store them in all kinds of formats across the repos.

So this is like a standard way. So it's actually a Jinja template plus this YAML at the top. So the YAML describes the metadata of the prompt, and then the Jinja template, you know, it's a template that you can pass things into.

Uh, so, uh, this is used by Promptflow, but it's also used by Azure AI Studio. And the goal is, and I think maybe LangChain might have support for it-

Mm-hmm

... now or soon. Uh, but the goal is just to have a common way of representing prompts. So we'll probably try to use this in more of our stuff going forward.

There you go. Just ask Copilot. [laughing] [laughing]

Yeah, so I'm using-- this is using the Promptflow evals, uh, package, um, which has a bunch more things as well, other kinds of evaluation. Uh, they actually-- I wrote my own CLI and UI on top of this, but they have, they have one too that you can use.

Yeah. Prompty.

Do you run the evals in your CI pipeline somewhere?

Yeah, if you look at Azure Dev, so this one, it does actually run them. Um, so I run them. Right now I'm just running them as a smoke test for this repo.

Um, but, uh, you can see what I've done is that, uh, I have a target URL. So that's generally what you'd wanna do is you, you need to run the eval against your, your, your live, where like for you, you're doing CI builds.

So there you wanna run it against your-

Yep

... CI build. So the tricky thing is just making sure you have a way of contacting your thing with everything, all the, you know, production setup, right, all the Azure stuff in it.

Uh, so, uh, yeah. I want it-- I would ideally have it as a CI step for every one of our repos, and I'm just figuring out the right way of setting up, like, the target URL and all that stuff.

Especially if you s-- if-- 'cause most people aren't making public facing apps. Most people are either putting it behind user auth or putting it in a VNet. So we need evaluation flows that both can use your production resources, because that's how you know it's working, uh, but that also works with however your app is deployed.

So I think you can certainly figure out how to set it up for your situation. Uh, I'm still figuring out how to set it up in the general case.

Uh, but the thing to keep in mind is that evaluations are slow if you're doing GPT metrics, right? Or, I mean, generally they're slow because s- all of these calls are slow.

You saw how much time-

Yeah

... it took to get back a response, right? So generally, they're slow. They're much slower than traditional unit tests. So you do not want to casually run an evaluation.

They're also expensive, especially if you're doing... Well, first, 'cause the LLM calls happen behind the scenes, and if you're using GPT metrics, 'cause I'm doing all these GPT metrics like relevance and groundedness, that's another LLM call.

So you wanna have, like, a higher barrier to running than with normal unit tests, right?

And-

So-

And caching

... caching. Oh, you cache.

Yeah.

Wait, how do you cache? How do you know that something hasn't changed?

Based on a prompt-

Oh, shoot

... and the test. Yeah.

Yeah, I guess, yeah, if you're... if it's all within one repo. This one is, like, a repo that works-

Yeah

... with other repos. You don't know if the app has changed behind the scenes.

Yeah.

Um, but yeah, if you, you have some caching. If you do caching, that's good.

Cache what exactly?

So we, we look at each test and we only rerun them when any of the prompts, when the input basically change. So if you imagine, like, an OpenAI proxy that you could set up if, if, if it's the same.

Similar, what, to what they do. I think OpenAI has, like, the seed variable-

Oh

... which is basically caching, but they don't tell you. Um, and it's basically if nothing changes in a prompt, it just sends back the old response.

Oh, so you implement caching in Copilot Chat, you mean? Or-

Not in Co- in, in our testing infrastructure

... Okay.

Yeah.

'Cause some people also implement caching within the RAG application itself.

Yeah.

Or in the, maybe the open context app or something.

Yeah.

I just don't know how often you're gonna get the same question.

For, like, tests it helps. For, for tests-

For tests

... oh, it's semi. Yeah.

Yeah.

Yeah.

Yeah.

Yeah.

I haven't, I haven't, like, toyed with that.

Yeah. So can this also... Like, is this just for OpenAI? Can this work with Mistral and all the other ones you mentioned in the beginning?

Could, yeah.

Uh, yeah. I mean, mine just... This one I just hit up a URL and get back the answer. Uh, so the URL is just of your deployed app.

Oh, sorry. I meant, like, the starter templates.

Oh, yeah. Good question. So with the starter templates, uh, right now they're all configured with OpenAI, and so you can swap out, like, different OpenAI models. So you can do v4.

But, uh, they don't work with the new, um, non-OpenAI models because we can't necessarily use the OpenAI SDK with them. I think there is actually a way to use the OpenAI SDK with them, but we're supposed to pretend we can't. [laughs]

Um, so, uh, there is this new SDK and I haven't, I haven't messed with it yet. I don't know if you have. But, uh, Azure AI Inference, have you seen it?

Um, I think this is the new unified SDK.

Oh.

And, um, yeah. So this is, this is what to use for everything that's not OpenAI. Oh, it says you can even do OpenAI.

So we might have to port to this. The thing I don't love about this is that this is Azure specific, 'cause right now we use the OpenAI SDK, which is, like, not Azure specific ex- exactly.

Um, and so it works with, like, Ollama and stuff. I don't know. So but we might end up porting for this. So if we ported to this, then probably we would just...

it would just work, um, with everything. So w- this is really new. Like, this came out at Build. So we just have to decide whether to port everything up to this, uh, uh, out to this, so that we can, um, use all the, all the modules, all the models.

That's true.

Yeah. Everything changes all the time. [laughs] Yeah, but we would also need to make the Bicep for it. That's... The, the other thing I haven't done is I haven't... Uh, 'cause I try to set up Bicep for everything.

So typically Bicep creates your Azure OpenAI instance. If you were using Mistral or Llama, you would want Bicep to... You'd probably want Bicep to create that as well. Um, and so that would be different Bicep as an addition.

So we'll probably end up adding it, 'cause ba- basically what you do is you go to the issue tracker, you file a request, and then if enough people ask for it, we're like, "Okay,

guess we're gonna do it then." [laughs]

Well, we can.

Yeah. [laughs]

We can.

Yeah. Um, but that's how we figure out what, you know, what it is that people are looking for. 'Cause it is really nice to be able to swap out models.

'Cause right now all of the samples do work with Ollama. So if you have, like, uh, Ollama running locally, here's my little Ollama up there. Um, you know, you can run, like, Phi-3 and stuff like that.

Uh, you just go... You know, you go to your terminal and you're like, "Ollama." Is this it? I think. I don't know if I s- typed Phi-3 correct, but...

Um, and so they do all run with, uh, Ollama things, but none of the Ollama models have really been sufficient for RAG in my experience. Like, I, I run them just to check, but they, they all fail to get, um, follow directions in my experience.

Uh, because I just think they're not... They don't have enough parameters. Like, these are, like, 3B, 7B, et cetera. So, um, they don't provide citations correctly. I don't know.

Have you had more success with, like, Phi-3 Mini?

We see every model requires prompt changes. Might be-

Yeah

... the way to go back to prompts

... and I'm bad at prompt engineering.

Maybe so.

Yeah. So, [laughs] so out of the gate, like, I haven't had success using any of the small language models for RAG. I'm sure the big versions of them would work much better.

So I do wanna try out, like, the 70B.

Maybe. Maybe 7B, yeah.

I've done, I've done up to 7B-

7B

... 'cause that's, like, how far I can go up locally. I can't go much more than that just from space reasons.

Yeah.

So for them, what happens is that, like, they'll answer the questions fine. The issue is that we need citations to be in good format, 'cause these are actually come back as bracketed square brackets, and they just don't reliably come back with square bracketed citations.

Which doesn't sound like a big deal, but, like, we're trying to make clickable citations here. Uh, so that's the issue I've had, is that I, I think they're fine at synthesizing the information, but they don't follow the syntax directions in terms of the citations.

Um, and they're kind of more, maybe more likely to, uh, make stuff up if I ask an off-topic question. That's been my experience there. Um-

They might be-

Yeah

... good for re-RAGging. I think Phi-3 is pretty small.

Oh.

Like re-RAGging

Yeah

... GPT judging. I think that, that that's where, like, finding the-- this one thing. Maybe not, like, the expert full answers that, that follows strict format, but, like, one of the smaller tasks.

But you can't-- So most of them don't support function calling off the bat. So you're doing-- Would you do re-ranking with just a simple... You'd have to figure out what syntax they come back with.

But they're all good at coding, so they can-

Yeah, that's right. That's true. Everyone's good at coding.

Yeah.

If you can turn something into a coding task-

Yeah

... you're good. [laughs] Yeah. That's another form of RAG is like, uh, I was telling someone last time, like, uh, these are, these are all, like, doing, like, kind of RAG on just a few documents at a time.

If you're trying to, like, analyze a whole database or, like, a huge number of documents, then you really want to, like, actually, uh, use like a SQL query, like, with, like, aggregate functions or do, like, a pandas query, right?

So at PyCon, we did a demo where you, like, upload a CSV, and then you, you say like, "Oh, I wanna count the top restaurants in it." And then it just comes up with the pandas code, and then it runs the pandas code in a sandbox environment.

So that's, that's another, like, in-I mean, increasingly common, uh, form of RAG, where if you want to, like, come up with insights and analysis and, and that sort of thing, then you wanna consider a different architecture where you're actually going to have the LLM generate pandas code or SQL code, it's very good at both of those, and

then run those in a safe way.

You know, what I've done for the, uh, syntax is I provided it a type string definition.

Okay.

And then the type string definition is- [clears throat] I don't try to do the syntax, I just have to give them the explicit answer that they wanna read for. And then now, yeah, type check.

Yeah. Do you-- Are you actually using TypeChat? 'Cause that's basically what TypeChat does.

Um, I essentially recreated what I-

Okay. [laughs] Okay. So yeah. But what you're describing, that's the same, the same approach.

Yeah. And I've seen it, it's a lot more better, like especially, uh,

Yeah. So that would be another... Yeah, when you lead-- We're trying. Um, I know Daniel's actually experimenting with Type-- Daniel's the creator of TypeChat. He is experimenting with TypeChat with the local models.

Um, so trying-- 'Cause we tried, we did also try TypeChat with Phi-3 locally to see if we could use it in-instead of function calling with OpenAI. And we were having a hard time with it.

Um, but I think Daniel, like, maybe has to, like, tweak the prompts, and maybe they'll end up working better, like-

I know the Phi-3 has a response format JSON.

I don't think it is. Maybe the bigger one, but not, not the smaller one.

Not the smaller one?

Yeah. 'Cause that was the, that was the funny part for me, is that I eventually had to just tell him, "Just try to give me the text. Just give me the JSON."

And it got so hard to do.

So you just told it to give you JSON?

Not for Phi-3, though. Give me 3, 5.

Yeah. Yeah. And that would work. Yeah. Yeah. That makes sense. Yeah. Yeah. With prompt, like, prompt engineering like that, then it will

work. Nice. Cool. All right, everyone. Well, you have the passes for seven days, so feel free to keep deploying. [laughs] Uh, if you have any feedback for the workshop, you can tell us, or we have a survey there, which I assume is anonymous.

Uh-

It is.

Yeah, it's anonymous. And so you can fill out that one as well. You can take a picture, fill it out later.

And that was it. Thanks. [applause] [outro jingle]