AI Engineer World's Fair 2024
Creating and scaling your own custom copilots with Azure AI Studio
Read the talk
Building a sales copilot you can trace, evaluate, and monitor
A sales assistant can retrieve the right data and still draw the wrong chart. Hanchi Wang’s Azure AI Studio walkthrough follows the engineering work needed to understand and improve that behavior.
From a talk by Hanchi Wang
Before you start: Basic familiarity with Python functions, SQL queries, and language-model tool calling will help you follow the implementation.
What does it take to add AI to an application?
How do you turn a language model into an application that uses your business data, calls the right tools, and keeps working after deployment? Hanchi Wang opens with the shift from discovering generative AI in 2023 to putting it to business use in 2024. That shift exposes an engineering problem: a model cannot simply be attached to an existing application and left to operate. It needs domain-specific knowledge and tools.
The integration also needs content-safety filtering, systematic output evaluation, and continuous monitoring after deployment. The application lifecycle extends beyond generating an answer. Azure AI Studio and Prompt flow are the two complementary tools Wang uses to work through that lifecycle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cloud hub and local developer tools
Azure AI Studio brings models, cognitive capabilities, search, and pretrained APIs into a development hub. Azure Machine Learning supplies the deeper data-science, model-training, and LLMOps capabilities, alongside tools for trustworthy and secure AI systems. In this 2024 walkthrough, Studio had reached general availability at Microsoft Build the previous month. Wang reports more than 20,000 monthly enterprise customers at the time.
Prompt flow supplies an SDK, CLI, and VS Code extension for building, testing, tracing, and evaluating applications. Its open-source tooling is free and can run locally without an Azure account; that does not make model inference or hosted services free. Connecting it to Studio adds cloud tracking for assets, traces, service connections, and run results.
The walkthrough concentrates on three connected jobs:
- Tracing and instrumentation: capture application events and inspect what happened inside a request.
- Evaluation: measure application quality and safety so changes can be compared.
- Monitoring: observe deployed workloads and investigate production behavior.
These jobs share evidence: a trace that explains a local failure can also help interpret an evaluation result or diagnose a deployed request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two tools, a follow-up question, and an incorrect chart
The running example is a chatbot for an outdoor-equipment company’s sales data. It uses the Assistants API to coordinate two tools. A custom tool translates a natural-language question into SQL and fetches the corresponding data. A built-in code interpreter works with that data to draw charts. The initial response combines a chart with an explanation of what the assistant did.
Wang first requests monthly sales for 2024 and a line chart. He then asks for a follow-up: overlay the bar chart with a line showing the percentage difference compared with 2023. This depends on conversational context—the user does not repeat the year, grouping, and sales-analysis task from scratch. The demonstration uses the historical Assistants API; its persisted conversation history supports this interaction, although later documentation clarifies that threads can be truncated to fit the model’s context window.
The assistant produces a line chart, but Wang identifies it as incorrect. He had encountered similar behavior while preparing and switches to a prepared correct result. In that result, the percentage differences are all negative: the comparison indicates declining sales relative to the previous year. Producing a chart is not the same as answering the question correctly. The failure makes evaluation and monitoring immediate requirements for this application.
The application’s sequence diagram separates the assistant’s orchestration from the SQL retrieval tool and code interpreter. Prompt flow wraps the application as an outer instrumentation layer, enabling tracing and monitoring without making those concerns the assistant’s business logic. Wang then points viewers to a GitHub repository containing the demonstration code before moving into the Python implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow the request from Python function to generated SQL
The chatbot’s public interface is an ordinary Python function: it receives a user question and returns chat output. The addition is Prompt flow’s @trace decorator, which captures function inputs, outputs, and execution events. The configured tracing also captures LLM interactions such as Azure OpenAI calls. Prompt flow’s tracing documentation distinguishes function decoration from initialization with start_trace() for LLM instrumentation; the decorator alone is not the entire setup.
The Sales Data Insight tool follows the same pattern as a callable Python class with a traced entry point. This keeps observability attached to meaningful application boundaries. For example, an adapter around an existing question-to-SQL function can preserve that callable structure:
python
from collections.abc import Callable
from promptflow.tracing import start_trace, trace
start_trace()
class SalesDataInsight:
def __init__(self, generate_sql: Callable[[str], str]):
self.generate_sql = generate_sql
@trace
def __call__(self, question: str) -> dict[str, str]:
sql = self.generate_sql(question)
return {"question": question, "sql": sql}
The wrapped function supplies the SQL-generation behavior; the decorator makes its input and returned SQL inspectable. In Wang’s application, the tool also retrieves the data. The second tool, Code Interpreter, is built into the Assistants API and does not require an equivalent custom implementation.
Prompt flow provides a local testing UI for exercising these entry points. Wang launches it with pf flow test, pointing the command at the chat-completion function, then repeats the sales question. While the request runs, he opens the trace view and refreshes it to see new traces arrive. The UI runs on his local machine, so this inspection is part of the development loop rather than a task deferred until deployment.
Once the request completes, the trace supplies a sequence of progressively more specific questions:
- Chat boundary: what question entered the application, and what answer came back?
- Tool boundary: how did the assistant rephrase the question before passing it to Sales Data Insight, and what did the tool return?
- LLM call: what SQL did the model generate from that question?
- Code Interpreter: what Python did the interpreter generate to produce the chart?
This separation matters for the earlier failure. A wrong chart could originate in question interpretation, SQL generation, or chart-producing code; the final image alone cannot locate the problem.
The emitted instrumentation uses OpenTelemetry traces and events. Collectors can route it to destinations such as Azure Monitor, and an application that already emits OpenTelemetry can inspect its existing spans alongside Prompt flow’s instrumentation. The result is a connected request history rather than a separate AI-only debugging record.
For collaboration, Wang has configured locally generated traces to reach Azure AI Studio. Studio presents the same trace view, with a URL he can share with a colleague and storage beyond the local testing session. The instrumentation is framework agnostic: the same decorator approach can cover a single LLM call, retrieval-augmented generation, function calling, or single- and multi-agent workflows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn sales questions into repeatable evaluations
Tracing explains an individual execution. Evaluation asks whether the application behaves well across a collection of inputs, including cases where an LLM fabricates data or returns an incorrect response. It also supplies evidence for choosing among models on accuracy, cost, and performance. Wang narrows the target to the Sales Data Insight tool, where natural-language questions become SQL queries.
Because model outputs are nondeterministic, evaluation needs to run regularly against a high-quality test dataset. Prompt flow offers tools for generating synthetic test data, but Wang uses a prepared dataset. Its question column contains what a user might ask the chatbot; its ground-truth column contains the expected output from the Sales Data Insight tool. This gives each generated query something concrete to be evaluated against.
The evaluate function connects three parts: the dataset, the target application or tool, and the evaluators. It acts as the execution engine, while each evaluator defines a particular check. Wang combines a built-in safety evaluator with three custom evaluators:
| Evaluator | What it checks |
|---|---|
| Content safety | Harmful, hateful, or violent content in queries and outputs |
| Execution time | Time taken by the target |
| Error rate | Frequency of errors |
| SQL similarity | Generated SQL’s similarity to ground truth |
Safety and task quality answer different questions. A response can pass a content-safety check while still containing the wrong SQL. Likewise, SQL similarity is the demonstrated comparison measure; it should not be silently treated as proof that two queries return equivalent results.
Wang opens results from an evaluation run completed before the talk, first in the terminal and then in Studio. Studio shows aggregate metrics at the top and individual dataset rows below. He describes the displayed safety results as satisfactory. For a surprising result, the page also exposes evaluation traces, letting the developer inspect how a metric was calculated instead of accepting only its aggregate value.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a model using the application’s own dataset
At the time of the talk, Wang reports more than 1,600 models in Studio’s catalog. Catalog benchmarks help with discovery, but the more relevant experiment is to run the same sales-query dataset through different models. His earlier runs include Cohere models and the Phi-3, Mistral, Llama, and GPT families. He selects GPT-35-Turbo, GPT-4 Turbo, and Mistral Large for a closer comparison, using GPT-35-Turbo as the baseline.
For Wang’s prepared natural-language-to-SQL dataset, the displayed runs have the following relative results:
| Model | Execution time | Error rate | SQL similarity |
|---|---|---|---|
| GPT-35-Turbo | Fastest of the three | Higher than the other two | Lowest of the three |
| Mistral Large | Slower than GPT-35-Turbo | Lower than GPT-35-Turbo | Higher than GPT-35-Turbo |
| GPT-4 Turbo | Slowest of the three | Same as Mistral Large | Highest of the three |
Wang therefore prefers GPT-4 Turbo for this application if speed is not a concern. The comparison is specific to his dataset and evaluators; he gives relative rankings without numerical scores or a cost comparison. The useful decision is the tradeoff: higher SQL similarity requires accepting more execution time in these runs.
The evaluation machinery does not require every metric to come from a platform-provided catalog. Custom evaluators can be ordinary Python functions, allowing application-specific checks to run alongside built-in evaluators such as content safety. That makes the evaluation target’s actual responsibilities—not just a model’s general benchmark performance—the basis for comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use production telemetry to decide what to fix
Deployment changes the scale of observation. Engineers still need the details of individual requests, but they also need system-wide telemetry to see whether the application continues to behave as expected. Wang’s team deploys the chatbot, sends Prompt flow’s OpenTelemetry spans and events to Azure Application Insights, and builds a dashboard over them.
The dashboard separates performance, usage, and failure metrics. In the displayed dashboard, GPT-4 Turbo has the highest model duration among the shown models, consistent with the earlier evaluation. Prompt tokens account for most of the displayed token usage, with comparatively few completion tokens. That distribution suggests a concrete cost investigation: examine whether the prompt templates can be shortened. It is a proposed intervention, not a demonstrated optimization.
Failure counts provide another entry point. Wang tentatively attributes some spikes to rehearsals earlier in the week. When a failure needs investigation, the developer can move from the dashboard into a specific request’s trace and stack trace. Aggregate monitoring identifies a pattern; request-level evidence helps explain its cause.
The closing product scope is broader than the sales demo: an end-to-end toolchain for efficient AI application development, with responsible AI practices, enterprise scalability, and security as stated platform priorities. In the application just examined, that lifecycle has a concrete shape: preserve the execution evidence, evaluate the SQL-generating tool against known examples, and keep inspecting real workloads after deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Microsoft's open-source Python tooling for developing, tracing, evaluating and deploying LLM applications.
Further reading
Examples of LLM instrumentation, function decorators and OpenTelemetry-compatible traces.
A historical sales-analysis example combining SQL retrieval, chart generation, local testing and cloud tracing. Setup uses older Azure APIs and model deployments.
Application Insights trace inspection and KQL dashboard examples for model duration and token usage.
Contemporary Microsoft recap confirming Studio general availability and a catalog exceeding 1,600 models.
Updates since the talk
- Azure OpenAI Assistants and conversation threadsDocumentation
Later documentation explaining Assistants, tools and persisted threads, including context-window truncation.
Read the complete timestamped transcript
- 0:00
[on hold music] Hello, everyone. My name is Hanchi Wang.
- 0:15
I'm a software engineer lead at, uh, Azure AI of Microsoft, and welcome to this talk. Um, if 2023 is the year when the world discovered generative AI, then 2024 is the year organizations truly began using and deriving business values from this new technology.
- 0:35
And, um, I can actually s- go quick on some of the slides because Cedric showed some of them already. But, uh, Azure at Micro- at Microsoft, Azure AI is, um, helping a lot of customers, and we definitely saw the AI adoption ex- accelerating.
- 0:53
Um, and our customers, enterprise customers are lar- landing their AI solutions. At the same time, many other companies are just starting to explore what AI can do for them.
- 1:04
And, um, but like as AI engineers, you and I know that AI is not just a plug-and-play solutions. Uh, it cannot be just tagged onto existing applications. Large language models need to integrate with domain-specific knowledge and tools.
- 1:21
O- outputs of these models must be carefully filtered for content safety and thoroughly evaluated. Moreover, once the AI application is deployed, engineers must continuously monitor its performance to ensure it operates ef- effectively.
- 1:37
Um, at Microsoft, my team has developed a suite of comprehensive tools that enable engineers to build their AI applications efficiently. And today, I'll introduce to you two of our products, Azure AI Studio and Prompt flow, and describe how they help engineers to build their own AI-powered applications.
- 1:59
Um, Cedric showed this a little bit already, um, and AI, Azure AI's portfolio is hosted on Azure AI Studio, the development hub for building AI applications. Within Studio, you'll find AI services like models, cognitive capabilities, and search, giving you pre-trained APIs so that you can get into production faster.
- 2:21
We offer Azure Machine Learning for advanced data science, model training, and full LLM ops management. We also have an end-to-end suite of tools to make AI systems trustworthy, safe, and secure.
- 2:36
And obviously, last month at Microsoft Build, Satya announced the general availability of Azure AI Studio, which is a big milestone. Uh, we now have more than twenty thousand monthly enterprise customers using the platform.
- 2:50
And Prompt flow, on the other hand, uh, complementing the Azure AI Studio, provides SDK, CLI, VS Code extension, and other developer tools to streamline the end-to-end development cycle of AI applications.
- 3:06
It helps developers to build, test, trace, and evaluate their applications with ease. Prompt flow is one hundred percent open source and free. You can use it purely in a local setup without the need of an Azure account, but it's, it's more powerful when used together with Azure AI Studio, where you can track your assets, traces, service connections,
- 3:28
and run results in the cloud. And, uh, Azure AI Studio and Prompt flow have so many amazing features. I'd like to highlight three of their superpowers in this talk.
- 3:43
Tracing and instrumentation allow developers to capture events from their applications and gain insights into what might have went wrong. Evaluation capabilities give you ways to improve your application for quality and safety.
- 3:59
And monitoring enables developers to be proactive and have visibility into production workloads.
- 4:06
To show you a real-world example, here's an application my colleagues and I have built using Azure AI Studio and Prompt flow.
- 4:17
Okay. Um, it's a basically a chatbot app that helps to answer sales data questions for a outdoor equipment company. I have some questions prepared here already
- 4:33
so that you don't need to see my awkward typing. Um, but this app is using the Assistant API, and Assistant API has access to two tools. The first tool knows how to translate a natural language question to SQL query and fetch data.
- 4:51
Uh, and the second tool is a code interpreter tool, which knows how to m- make use of that data and draw charts, right? And so this is the result of the first question, and you can see it helps to...
- 5:06
It, like it draw the chart and also explained what it is doing. And we can also ask a follow-up question like- It's not on the screen. Oh. Oh, okay.
- 5:17
Sorry about that. What's going on? Um.
- 5:42
Do I need to duplicate?
- 5:43
Oh, it's either on mirror or on extended mode. If it's on extended mode, you would have to move your item over to that screen, yeah.
- 5:50
Oh, okay. Maybe that will be easier. Where is that? Okay.
- 5:56
Yeah. You could do duplicate as well.
- 5:58
Okay. I'll do this. Um, yeah
- 6:10
Name display here. Here's your, here's your mouse. Um, name display.
- 6:23
And mirror
- 6:23
Mirror?
- 6:23
Yeah.
- 6:26
Okay. Okay, cool. And do I switch back once I go back to my slides?
- 6:32
No, because it's just mirroring from here to here.
- 6:35
Okay.
- 6:35
Yeah.
- 6:35
Okay, cool. That's good. Thank you. Cool. Um, so yeah, you can see that the question I asked is the, "Show the sales data in twenty twenty-four by month," and ask it to draw a line chart, right?
- 6:45
And it, it just did that. And I can ask a follow-up question like, uh, overlay that, the bar chart with a line showing the percentage difference comparing to twenty twenty-three.
- 6:58
And what's cool about the Assistant API is that it always has the chat history in the context, so that I don't need to restate the previous question. It has, like, everything, um, in the context ready so that it hopefully will be able to, uh, like, answer that in a little bit.
- 7:19
Okay. It's thinking about this. Okay. Um, this, um, uh, it did draw a line chart, but, like, it's incorrect.
- 7:37
And, um, it does happen from in my preparation, so I had a backup plan, which is like, um, this is actually the line chart. And that's why actually it is important to do evaluation monitoring, which is something I will show next, right?
- 7:51
But this is the correct line chart. Um, unfortunately, the numbers are all negative, so it looks like this company is not very doing-- uh, not doing very hot currently, unfortunately.
- 8:02
Um, okay. Now going back to my slides. Okay, cool.
- 8:14
And here's the sequence diagram of the app. With the Assistant API, it has access to, to two tools. The first one knows how to fetch data using the query generated from a natural language question, and the second tool is a code interpreter.
- 8:28
And by adding Prompt Flow as a alt layer of the application, it enables superpowers like tracing, monitoring, as we will see.
- 8:40
Okay. Uh, before I show you the code, uh, can you guys see this link? You can write a little bit. Yeah, so all the code I will show, you can find it from this GitHub repository, and you can, like, uh, clone it and try it out yourselves.
- 8:58
Tracing and the instrumentation are top priorities for developers. Let's face it, the code we developers write doesn't always run the way we expect, right? Um, so that's why it's critical for developers to understand how the code really runs.
- 9:11
Let me show you the code of the chatbot app now.
- 9:15
If I... Okay. Oh. Um. Okay, you don't see it, right?
- 9:34
Hmm. It's still in... Okay. Um... Okay. So this is the public interface of the chatbot, and it's just a typical function.
- 9:50
It takes a question from the user and returns a chat output here. The only thing special here is the trace decorator. Oh, sorry about that. Provided by, um, Prompt Flow.
- 10:01
And the trace decorator helps, um, to emit, uh, events, uh, emit traces to capture inputs, outputs, and events of a Python function, and it also automatically captures any interactions with LLMs, like Azure OpenAI.
- 10:20
And I can also quickly show you the code for the sales data insight tool.
- 10:26
Again, it's nothing but a typical callable Python class with the trace decorator here.
- 10:33
And with the, um, and f- the second tool is a code interpreter, which is the built-in tool from the Assistant API, so there's no code for that. And, uh, as a engineer, you might be wondering what is the best way to test this application and look at traces.
- 10:51
Prompt Flow provides a local UI just for that. Okay.
- 10:57
And to... Okay. To start the UI, this is a command you would run, and it pf flow test, and you point at, point it at the chat completion function we were looking at earlier.
- 11:17
And I have already ran the command earlier, so, um, here is the UI. You can see it's running purely on my local machine, and we can ask it a question.
- 11:31
Like... Oops. Like the same question we were asking the chatbot app.
- 11:43
While this, while this is running, I can look at the traces.
- 11:50
Already that is coming up. It's still thinking about it, so I can click refresh to see the upcoming traces.
- 12:01
Um, now looks like it has completed. Let's dig into the trace a little bit.
- 12:07
From the high level, this is the-
- 12:10
Out layer of the chatbot app, you can see the input and output from the chat. And, uh, we can look, dig into the sales data insight tool a little bit and see that the assistant API actually modified the question a little bit, and the, the tool was able to reply the right output.
- 12:31
And if we really want to understand how the LLM was able to do that, we can drill down even one layer deeper to see, oh, this is the question from the customer, and this is the auto-generated SQL query.
- 12:47
And lastly, this is the code interpreter tool which writes some Python function and eventually draw the chart we were looking at
- 12:59
here. Okay. And what, what do I should point out is that all the instrumentation emitted by Prompt flow are open s- OpenTelemetry's traces and events, uh, and OpenTelemetry is obviously, uh, industry standard.
- 13:15
That means you can configure your collectors to send the traces to your preferred destination, like Error Monitor. And it also means that if you have other part of your application already sending OpenTelemetry traces, you can look at that and the traces sent by Prompt flow at the single place.
- 13:36
And, um, you may also wondering, like if I want to like collaborate with a colleague, how do I do that on a local UI, right? And Azure AI Studio has that covered.
- 13:48
Uh, in my local environment, I have configured my
- 13:53
traces to go to the, um, Azure AI Studio already, and that's how I got to this page. And you can see it's the same trace view on Azure AI Studio, and I can s- simply copy and paste this URL and share with, share it with someone.
- 14:13
Okay. Coming back to this. Yeah, that wraps up the first superpower, tracing. Uh, the trace decorator can be added to any apps from single LLM calls to RAG, function calling, or single/multi-work, uh, agent workflows, and is framework agnostic.
- 14:37
Prompt flow provides a local UI for testing with trace views. To keep the traces for longer time or share with a colleague, I can view the traces in Azure AI Studio.
- 14:50
Now let's talk about evaluation, focusing on application quality. LLM-based application can be unpredictable, uh, in map fabric data or give incorrect responses, impacting the application quality.
- 15:04
Um, and o- often, like, folks may want to, like, try out different models, compare the pros and cons, and see which one is the best for their application in terms of accuracy, cost, or performance.
- 15:17
Um, and if you recall, the... A lot of the magic in the chatbot app happened in a single tool called Sales Data Insight tool, and that was able to kind of translate natural language questions into SQL queries using LLMs.
- 15:30
Let's see how we can evaluate that. Okay.
- 15:39
Okay. Cool. So the... Okay. Okay.
- 15:55
Um, due to the non-deterministic nature of LLMs, it's, running evaluation regularly is definitely critical, and what's im- equally important is to have high-quality test dataset. If you don't, um, Prompt flow has a suite of tools to help you to generate a synthetic test dataset.
- 16:15
In my case here, I have already prepared a test dataset, and the question column is what a user would ask the chatbot app, and the ground truth column is what the sales data insight tool would generate based on that question.
- 16:34
Okay? Let's also take a look at the evaluation code.
- 16:37
I'm using the evaluate function provided by Prompt flow for the evaluation. Uh, sorry, I'm using the evaluate function, um, provided by Prompt flow
- 16:49
from this namespace for the, um, evaluation, evaluation logic here. And the evaluate function, you can consider that as an execution engine that links the test dataset and the target, evaluation target and evaluators together.
- 17:09
I like to focus on the evaluators here. The first evaluator here is a content safety evaluator that comes with the Prompt flow SDK. It helps to act as a safeguard to make sure the query and the generated output are free of any harmful, hateful, or, um, uh, violent content.
- 17:30
And the evaluate-- the three evaluators following are custom evaluators, which gave me execution time, error rate, and sync- SQL similarity scores. The SQL similarity scores is calculated based on how similar the generated SQL query is to the ground truth.
- 17:49
Okay? I have pre-- Uh, I have ran a evaluation previously, uh, before this talk, and I can look at the evaluation results briefly from the terminal.
- 18:07
But what is even better is that I can look at the evaluation result on Azure AI Studio for a better view.[background chatter]
- 18:27
Okay. I can see the aggregated metrics at the top and... Okay. [chuckles]
- 18:37
If I refresh. Okay. Uh, I can see the aggregated re-results at the top. I can see my content safety metrics here, looks like I'm golden for that, and per row level result below.
- 18:50
What's really cool about this page is that I can also look at traces for the evaluation, in case I wonder how some of the metrics might have been calculated.
- 19:04
Okay. Uh, Cedric, in his talk, showed the model catalog, so I don't need to, like, talk too much about that. But another scenario that, um, evaluation really shines is to compare across different models, right?
- 19:19
Obviously, like you- there are, like, more than sixteen hundred models on Azure, um, AI Studio for the model catalog, and I can look at, like, benchmarks, right? But what is even better, like, in my case, is that I have a custom data set, and I want to evaluate the same data set across different models.
- 19:40
I did just that earlier this week, and here is the result. I found a couple of popular models on the model catalog. Uh, you can see the Cohere here, three...
- 19:51
Phi-3, Mistral, uh, Llama, and GPT, and I, uh, y- in a list view.
- 20:02
I can pick a couple of them to compare. For example, I can pick the GPT-35-Turbo, GPT-4 Turbo, and maybe Mistral Large.
- 20:15
Okay. In this comparison view, you can see that if I, if we use the GPT-35-Turbo as the baseline,
- 20:22
um, Mistral Large has higher execution time, which means it's slower, but it has better error rate and higher SQL similarity score.
- 20:33
If we put GPT-4 Turbo into picture, GPT-4 Turbo has the highest execution time, which means it's the slowest among the three. It has the same error rate compared to Mistral Large, and it has the highest SQL sim- SQL similarity score.
- 20:48
So this is telling me that if speed is not a concern, then GPT-4 Turbo is still the best choice for my situation.
- 21:00
Okay. With the second, uh, superpower, I just showed you the evaluate function and, uh, the built-in evaluators like the content safety evaluator, and also you can write your own custom evaluators using Python function.
- 21:23
For the final superpower, I like to concentrate on moving to production. Uh, application devel- deployment does not mean the job is done for an engineer. On the contrary, it's the start of a new journey.
- 21:35
It's a developer's responsibility to continuously monitor the application to ensure the application always runs as expected, and the developer needs to look at, uh, both the details for each individual request and also the overall telemetry for the whole system.
- 21:54
And Prompt flow, uh, tracing sends open telemetry spans and events which can be collected by multiple tools, including Azure Application Insights. We deployed the chatbot app and started to collect traces in our Application Insights resource and built a dashboard.
- 22:13
Here is the dashboard. Oops, sorry. Okay.
- 22:28
Um, you can see there are performance metrics, usage metrics, and failure metrics on this dashboard. And there's so many interesting insights we can draw from here. I'll just highlight a couple.
- 22:39
Uh, for example, the model duration is, like, basically the speed of the model, right? And GPT-4 Turbo has the highest number compared to other models, which aligns with the evaluation result we were looking at earlier.
- 22:53
And for the token usage, we can see majority of the tokens are actually used for the prompt, uh, and only a few are for the completion. So if we are concerned about the cost, then maybe we need to, uh, short the prompt templates a little bit.
- 23:10
And the failure tells me, like, how many of the calls failed, and some of the spikes may come from, um, some of the rehearsal I did earlier this week.
- 23:19
Uh, and if something really went wrong, we can also, like, dig into a specific trace to look at the same trace view and the stack trace.
- 23:28
Okay. And final slide. Uh, this is just a glimpse of what's possible with Azure AI Studio and the Prompt Flow. My team at Azure AI is dedicated to provide an end-to-end suite of tools to make AI application building easy and efficient.
- 23:49
At the same time, we also always have responsible AI in mind and provide enterprise-grade, uh, scalability and security. I can't wait to see what you do with Azure AI Studio and Prompt Flow.
- 24:01
Thank you all. [outro music]