Creating and scaling your own custom copilots with Azure AI Studio

Talk transcript

1. 0:00

   [on hold music] Hello, everyone. My name is Hanchi Wang.

2. 0:15

   I'm a software engineer lead at, uh, Azure AI of Microsoft, and welcome to this talk. Um, if 2023 is the year when the world discovered generative AI, then 2024 is the year organizations truly began using and deriving business values from this new technology.

3. 0:35

   And, um, I can actually s- go quick on some of the slides because Cedric showed some of them already. But, uh, Azure at Micro- at Microsoft, Azure AI is, um, helping a lot of customers, and we definitely saw the AI adoption ex- accelerating.

4. 0:53

   Um, and our customers, enterprise customers are lar- landing their AI solutions. At the same time, many other companies are just starting to explore what AI can do for them.

5. 1:04

   And, um, but like as AI engineers, you and I know that AI is not just a plug-and-play solutions. Uh, it cannot be just tagged onto existing applications. Large language models need to integrate with domain-specific knowledge and tools.

6. 1:21

   O- outputs of these models must be carefully filtered for content safety and thoroughly evaluated. Moreover, once the AI application is deployed, engineers must continuously monitor its performance to ensure it operates ef- effectively.

7. 1:37

   Um, at Microsoft, my team has developed a suite of comprehensive tools that enable engineers to build their AI applications efficiently. And today, I'll introduce to you two of our products, Azure AI Studio and Prompt flow, and describe how they help engineers to build their own AI-powered applications.

8. 1:59

   Um, Cedric showed this a little bit already, um, and AI, Azure AI's portfolio is hosted on Azure AI Studio, the development hub for building AI applications. Within Studio, you'll find AI services like models, cognitive capabilities, and search, giving you pre-trained APIs so that you can get into production faster.

9. 2:21

   We offer Azure Machine Learning for advanced data science, model training, and full LLM ops management. We also have an end-to-end suite of tools to make AI systems trustworthy, safe, and secure.

10. 2:36

    And obviously, last month at Microsoft Build, Satya announced the general availability of Azure AI Studio, which is a big milestone. Uh, we now have more than twenty thousand monthly enterprise customers using the platform.

11. 2:50

    And Prompt flow, on the other hand, uh, complementing the Azure AI Studio, provides SDK, CLI, VS Code extension, and other developer tools to streamline the end-to-end development cycle of AI applications.

12. 3:06

    It helps developers to build, test, trace, and evaluate their applications with ease. Prompt flow is one hundred percent open source and free. You can use it purely in a local setup without the need of an Azure account, but it's, it's more powerful when used together with Azure AI Studio, where you can track your assets, traces, service connections,

13. 3:28

    and run results in the cloud. And, uh, Azure AI Studio and Prompt flow have so many amazing features. I'd like to highlight three of their superpowers in this talk.

14. 3:43

    Tracing and instrumentation allow developers to capture events from their applications and gain insights into what might have went wrong. Evaluation capabilities give you ways to improve your application for quality and safety.

15. 3:59

    And monitoring enables developers to be proactive and have visibility into production workloads.

16. 4:06

    To show you a real-world example, here's an application my colleagues and I have built using Azure AI Studio and Prompt flow.

17. 4:17

    Okay. Um, it's a basically a chatbot app that helps to answer sales data questions for a outdoor equipment company. I have some questions prepared here already

18. 4:33

    so that you don't need to see my awkward typing. Um, but this app is using the Assistant API, and Assistant API has access to two tools. The first tool knows how to translate a natural language question to SQL query and fetch data.

19. 4:51

    Uh, and the second tool is a code interpreter tool, which knows how to m- make use of that data and draw charts, right? And so this is the result of the first question, and you can see it helps to...

20. 5:06

    It, like it draw the chart and also explained what it is doing. And we can also ask a follow-up question like- It's not on the screen. Oh. Oh, okay.

21. 5:17

    Sorry about that. What's going on? Um.

22. 5:42

    Do I need to duplicate?

23. 5:43

    Oh, it's either on mirror or on extended mode. If it's on extended mode, you would have to move your item over to that screen, yeah.

24. 5:50

    Oh, okay. Maybe that will be easier. Where is that? Okay.

25. 5:56

    Yeah. You could do duplicate as well.

26. 5:58

    Okay. I'll do this. Um, yeah

27. 6:10

    Name display here. Here's your, here's your mouse. Um, name display.

28. 6:23

    And mirror

29. 6:23

    Mirror?

30. 6:23

    Yeah.

31. 6:26

    Okay. Okay, cool. And do I switch back once I go back to my slides?

32. 6:32

    No, because it's just mirroring from here to here.

33. 6:35

    Okay.

34. 6:35

    Yeah.

35. 6:35

    Okay, cool. That's good. Thank you. Cool. Um, so yeah, you can see that the question I asked is the, "Show the sales data in twenty twenty-four by month," and ask it to draw a line chart, right?

36. 6:45

    And it, it just did that. And I can ask a follow-up question like, uh, overlay that, the bar chart with a line showing the percentage difference comparing to twenty twenty-three.

37. 6:58

    And what's cool about the Assistant API is that it always has the chat history in the context, so that I don't need to restate the previous question. It has, like, everything, um, in the context ready so that it hopefully will be able to, uh, like, answer that in a little bit.

38. 7:19

    Okay. It's thinking about this. Okay. Um, this, um, uh, it did draw a line chart, but, like, it's incorrect.

39. 7:37

    And, um, it does happen from in my preparation, so I had a backup plan, which is like, um, this is actually the line chart. And that's why actually it is important to do evaluation monitoring, which is something I will show next, right?

40. 7:51

    But this is the correct line chart. Um, unfortunately, the numbers are all negative, so it looks like this company is not very doing-- uh, not doing very hot currently, unfortunately.

41. 8:02

    Um, okay. Now going back to my slides. Okay, cool.

42. 8:14

    And here's the sequence diagram of the app. With the Assistant API, it has access to, to two tools. The first one knows how to fetch data using the query generated from a natural language question, and the second tool is a code interpreter.

43. 8:28

    And by adding Prompt Flow as a alt layer of the application, it enables superpowers like tracing, monitoring, as we will see.

44. 8:40

    Okay. Uh, before I show you the code, uh, can you guys see this link? You can write a little bit. Yeah, so all the code I will show, you can find it from this GitHub repository, and you can, like, uh, clone it and try it out yourselves.

45. 8:58

    Tracing and the instrumentation are top priorities for developers. Let's face it, the code we developers write doesn't always run the way we expect, right? Um, so that's why it's critical for developers to understand how the code really runs.

46. 9:11

    Let me show you the code of the chatbot app now.

47. 9:15

    If I... Okay. Oh. Um. Okay, you don't see it, right?

48. 9:34

    Hmm. It's still in... Okay. Um... Okay. So this is the public interface of the chatbot, and it's just a typical function.

49. 9:50

    It takes a question from the user and returns a chat output here. The only thing special here is the trace decorator. Oh, sorry about that. Provided by, um, Prompt Flow.

50. 10:01

    And the trace decorator helps, um, to emit, uh, events, uh, emit traces to capture inputs, outputs, and events of a Python function, and it also automatically captures any interactions with LLMs, like Azure OpenAI.

51. 10:20

    And I can also quickly show you the code for the sales data insight tool.

52. 10:26

    Again, it's nothing but a typical callable Python class with the trace decorator here.

53. 10:33

    And with the, um, and f- the second tool is a code interpreter, which is the built-in tool from the Assistant API, so there's no code for that. And, uh, as a engineer, you might be wondering what is the best way to test this application and look at traces.

54. 10:51

    Prompt Flow provides a local UI just for that. Okay.

55. 10:57

    And to... Okay. To start the UI, this is a command you would run, and it pf flow test, and you point at, point it at the chat completion function we were looking at earlier.

56. 11:17

    And I have already ran the command earlier, so, um, here is the UI. You can see it's running purely on my local machine, and we can ask it a question.

57. 11:31

    Like... Oops. Like the same question we were asking the chatbot app.

58. 11:43

    While this, while this is running, I can look at the traces.

59. 11:50

    Already that is coming up. It's still thinking about it, so I can click refresh to see the upcoming traces.

60. 12:01

    Um, now looks like it has completed. Let's dig into the trace a little bit.

61. 12:07

    From the high level, this is the-

62. 12:10

    Out layer of the chatbot app, you can see the input and output from the chat. And, uh, we can look, dig into the sales data insight tool a little bit and see that the assistant API actually modified the question a little bit, and the, the tool was able to reply the right output.

63. 12:31

    And if we really want to understand how the LLM was able to do that, we can drill down even one layer deeper to see, oh, this is the question from the customer, and this is the auto-generated SQL query.

64. 12:47

    And lastly, this is the code interpreter tool which writes some Python function and eventually draw the chart we were looking at

65. 12:59

    here. Okay. And what, what do I should point out is that all the instrumentation emitted by Prompt flow are open s- OpenTelemetry's traces and events, uh, and OpenTelemetry is obviously, uh, industry standard.

66. 13:15

    That means you can configure your collectors to send the traces to your preferred destination, like Error Monitor. And it also means that if you have other part of your application already sending OpenTelemetry traces, you can look at that and the traces sent by Prompt flow at the single place.

67. 13:36

    And, um, you may also wondering, like if I want to like collaborate with a colleague, how do I do that on a local UI, right? And Azure AI Studio has that covered.

68. 13:48

    Uh, in my local environment, I have configured my

69. 13:53

    traces to go to the, um, Azure AI Studio already, and that's how I got to this page. And you can see it's the same trace view on Azure AI Studio, and I can s- simply copy and paste this URL and share with, share it with someone.

70. 14:13

    Okay. Coming back to this. Yeah, that wraps up the first superpower, tracing. Uh, the trace decorator can be added to any apps from single LLM calls to RAG, function calling, or single/multi-work, uh, agent workflows, and is framework agnostic.

71. 14:37

    Prompt flow provides a local UI for testing with trace views. To keep the traces for longer time or share with a colleague, I can view the traces in Azure AI Studio.

72. 14:50

    Now let's talk about evaluation, focusing on application quality. LLM-based application can be unpredictable, uh, in map fabric data or give incorrect responses, impacting the application quality.

73. 15:04

    Um, and o- often, like, folks may want to, like, try out different models, compare the pros and cons, and see which one is the best for their application in terms of accuracy, cost, or performance.

74. 15:17

    Um, and if you recall, the... A lot of the magic in the chatbot app happened in a single tool called Sales Data Insight tool, and that was able to kind of translate natural language questions into SQL queries using LLMs.

75. 15:30

    Let's see how we can evaluate that. Okay.

76. 15:39

    Okay. Cool. So the... Okay. Okay.

77. 15:55

    Um, due to the non-deterministic nature of LLMs, it's, running evaluation regularly is definitely critical, and what's im- equally important is to have high-quality test dataset. If you don't, um, Prompt flow has a suite of tools to help you to generate a synthetic test dataset.

78. 16:15

    In my case here, I have already prepared a test dataset, and the question column is what a user would ask the chatbot app, and the ground truth column is what the sales data insight tool would generate based on that question.

79. 16:34

    Okay? Let's also take a look at the evaluation code.

80. 16:37

    I'm using the evaluate function provided by Prompt flow for the evaluation. Uh, sorry, I'm using the evaluate function, um, provided by Prompt flow

81. 16:49

    from this namespace for the, um, evaluation, evaluation logic here. And the evaluate function, you can consider that as an execution engine that links the test dataset and the target, evaluation target and evaluators together.

82. 17:09

    I like to focus on the evaluators here. The first evaluator here is a content safety evaluator that comes with the Prompt flow SDK. It helps to act as a safeguard to make sure the query and the generated output are free of any harmful, hateful, or, um, uh, violent content.

83. 17:30

    And the evaluate-- the three evaluators following are custom evaluators, which gave me execution time, error rate, and sync- SQL similarity scores. The SQL similarity scores is calculated based on how similar the generated SQL query is to the ground truth.

84. 17:49

    Okay? I have pre-- Uh, I have ran a evaluation previously, uh, before this talk, and I can look at the evaluation results briefly from the terminal.

85. 18:07

    But what is even better is that I can look at the evaluation result on Azure AI Studio for a better view.[background chatter]

86. 18:27

    Okay. I can see the aggregated metrics at the top and... Okay. [chuckles]

87. 18:37

    If I refresh. Okay. Uh, I can see the aggregated re-results at the top. I can see my content safety metrics here, looks like I'm golden for that, and per row level result below.

88. 18:50

    What's really cool about this page is that I can also look at traces for the evaluation, in case I wonder how some of the metrics might have been calculated.

89. 19:04

    Okay. Uh, Cedric, in his talk, showed the model catalog, so I don't need to, like, talk too much about that. But another scenario that, um, evaluation really shines is to compare across different models, right?

90. 19:19

    Obviously, like you- there are, like, more than sixteen hundred models on Azure, um, AI Studio for the model catalog, and I can look at, like, benchmarks, right? But what is even better, like, in my case, is that I have a custom data set, and I want to evaluate the same data set across different models.

91. 19:40

    I did just that earlier this week, and here is the result. I found a couple of popular models on the model catalog. Uh, you can see the Cohere here, three...

92. 19:51

    Phi-3, Mistral, uh, Llama, and GPT, and I, uh, y- in a list view.

93. 20:02

    I can pick a couple of them to compare. For example, I can pick the GPT-35-Turbo, GPT-4 Turbo, and maybe Mistral Large.

94. 20:15

    Okay. In this comparison view, you can see that if I, if we use the GPT-35-Turbo as the baseline,

95. 20:22

    um, Mistral Large has higher execution time, which means it's slower, but it has better error rate and higher SQL similarity score.

96. 20:33

    If we put GPT-4 Turbo into picture, GPT-4 Turbo has the highest execution time, which means it's the slowest among the three. It has the same error rate compared to Mistral Large, and it has the highest SQL sim- SQL similarity score.

97. 20:48

    So this is telling me that if speed is not a concern, then GPT-4 Turbo is still the best choice for my situation.

98. 21:00

    Okay. With the second, uh, superpower, I just showed you the evaluate function and, uh, the built-in evaluators like the content safety evaluator, and also you can write your own custom evaluators using Python function.

99. 21:23

    For the final superpower, I like to concentrate on moving to production. Uh, application devel- deployment does not mean the job is done for an engineer. On the contrary, it's the start of a new journey.

100. 21:35

     It's a developer's responsibility to continuously monitor the application to ensure the application always runs as expected, and the developer needs to look at, uh, both the details for each individual request and also the overall telemetry for the whole system.

101. 21:54

     And Prompt flow, uh, tracing sends open telemetry spans and events which can be collected by multiple tools, including Azure Application Insights. We deployed the chatbot app and started to collect traces in our Application Insights resource and built a dashboard.

102. 22:13

     Here is the dashboard. Oops, sorry. Okay.

103. 22:28

     Um, you can see there are performance metrics, usage metrics, and failure metrics on this dashboard. And there's so many interesting insights we can draw from here. I'll just highlight a couple.

104. 22:39

     Uh, for example, the model duration is, like, basically the speed of the model, right? And GPT-4 Turbo has the highest number compared to other models, which aligns with the evaluation result we were looking at earlier.

105. 22:53

     And for the token usage, we can see majority of the tokens are actually used for the prompt, uh, and only a few are for the completion. So if we are concerned about the cost, then maybe we need to, uh, short the prompt templates a little bit.

106. 23:10

     And the failure tells me, like, how many of the calls failed, and some of the spikes may come from, um, some of the rehearsal I did earlier this week.

107. 23:19

     Uh, and if something really went wrong, we can also, like, dig into a specific trace to look at the same trace view and the stack trace.

108. 23:28

     Okay. And final slide. Uh, this is just a glimpse of what's possible with Azure AI Studio and the Prompt Flow. My team at Azure AI is dedicated to provide an end-to-end suite of tools to make AI application building easy and efficient.

109. 23:49

     At the same time, we also always have responsible AI in mind and provide enterprise-grade, uh, scalability and security. I can't wait to see what you do with Azure AI Studio and Prompt Flow.

110. 24:01

     Thank you all. [outro music]