Multi-model, multimodal, and multi-agent innovations in Azure AI

Talk transcript

1. 0:00

   [on hold music] Uh, so, um, I'm Cédric Vidal.

2. 0:16

   I'm an, uh, principal AI advocate at Microsoft. Uh, and today, uh, I'm going to, uh... We're going to do quite cool stuff. Uh, we're going to do-- to talk about multi many things.

3. 0:29

   Uh, multi, uh, models, multimodality, multilingual, multi-agents. Uh, all of this with, uh, Azure AI. Um, so yeah. And also one particularity is, uh, apart from a few slides at the beginning, it's only demos.

4. 0:45

   Uh, and to be honest, bear with me in case, uh, one of them or all of them don't work. [laughs] Uh, but, uh, it's gonna be fun. We'll see how it goes.

5. 0:55

   Um, so as you know, you know, Azure AI is, uh, the best AI platform, uh, out there. Uh, we have, um, a lot of, uh, AI, uh, services. Uh, we can do, uh, machine learning.

6. 1:09

   Uh, and we also, uh, do all of that responsibly with the whole responsible AI framework. And we encapsulate all of this, um, in the Azure AI Studio, and I'm going to do, uh, a lot of demos, uh, of i-- Azure AI Studio today.

7. 1:28

   And, um, since now almost a year, a bit more than a year, um, we've been, uh, partnering with OpenAI, of course. Uh, and we have all of the OpenAI models available on Azure on the Azure platform.

8. 1:41

   But, uh, we're gonna see that, uh, in addition to all the Mo-- OpenAI models that we have and all the modalities that we can get using those, we also have many more models available o-on the platform for, um, uh, text of course, uh, vision, uh, and speech.

9. 2:01

   Um, and many organizations trust us today, uh, to, uh, use, uh, AI and build their products. Um,

10. 2:11

    so without, um, further ado. So, uh, I'm going to jump in the demos very quickly. But before I do, so we've had many things announced at Build a couple of months ago.

11. 2:24

    Uh, we've had, uh, the GA version of Azure AI Studio. We've had, uh, the latest, uh, model from OpenAI, GPT-4, uh, Omni, uh, which supports, uh, text, vision, uh, and soon speech.

12. 2:37

    Uh, we've had the new small la-language models, um, from Microsoft, from MS Research called Phi-3. Um, now we have also announced GPT-4 Turbo with Vision, uh, DALL·E 3, and Whisper.

13. 2:51

    Um, we've announced the Assistants, uh, API that allows to, uh, build, uh, agents, and I'm going to demo it. Uh, we've announced the fine-tuning for GPT-4, the new, uh, inference batch API.

14. 3:06

    Um, and um, also another very cool thing I'm going to demo today, and you had a glimpse of it. Uh, I mean, I guess the surprise is kind of, um, uh, out, uh, but the video, video translation service that I'm going to demo.

15. 3:21

    Um, and Azure AI Studio. Oh, yeah. So let's go straight to the demos now. Um, so apart from those slides, now it's only demos.

16. 3:32

    So the fun begins. Um, okay. The first demo. So we've had, um, Azure. I mean, uh, uh, like a year ago when, uh, everything started, uh, you didn't have much, uh, many choices.

17. 3:48

    It was basically, uh, GPT or, or GPT. The, the only modality available was text. Um, but now things have changed dramatically. Now, we support also, uh, multimodal vision, mixing text and vision, and this opens a completely new era of use cases.

18. 4:07

    For example, here and, um, let me zoom. So I'm going to demo, uh, GPT-4, um, o. And actually I selected Vision here, but what I wanted to select was GPT-4o.

19. 4:20

    Um, and I'm going to demonstrate a use case where... So it is kind of small right now, but, uh, this is, um, a menu from a restaurant. Um, and I'm going to ask what's vegan on the menu today.

20. 4:38

    So here we can see the menu a bit better.

21. 4:42

    So as you can see, we have, uh, winter chicory salad, uh, duck, sea bass, et cetera. Um, and what's very interesting here is that, uh, the menu that you just saw was, um, the, the font is funny, but it's printed.

22. 4:58

    Uh, so it's a font from a computer. Uh, and GPT-4o does a very good job at reading what's on the menu. Um, and let me zoom here so that we can see what's...

23. 5:11

    So I asked whether there were vegan options today on the menu. And what's interesting is that it looks, it mixes vision, so it extracted all the text from the image.

24. 5:23

    But not, not only that, but it reasons on it. So it analyzed all the items on the menu today and for each one of them, uh, looked at which ones were, uh, vegan.

25. 5:33

    And as you can see here, uh, the cauliflower soup and winter, um, uh, chicory salad. Uh, let me zoom. Ah.

26. 5:44

    Up. Okay. So, okay. Both mention, uh, vege-vegetarian and vegan versions. Um, so that's a very good example of how to mix, uh, text and reasoning.

27. 6:00

    Something that was not possible before with just OCRs, which becomes available with the new generation of multimodal, um, models.

28. 6:10

    Another example, slightly harder because this one has

29. 6:20

    handwritten text. So this menu has not been printed. It has been written by hand on a chalkboard.

30. 6:33

    So as you can see... Oh, and it's in French. Um, so not only does it recognize handwritten, um, uh, sentences written on a chalkboard in a picture, but it also translates it and reasons on it.

31. 6:54

    So that's three things that the model is doing all at once, thanks to, uh, multimodality. Uh, this is very important to understand how it differs from what we were doing before, because before we were using image to text to extract the text and then reason on it.

32. 7:12

    Now, the model understands natively both pixels and text, and in its internal, um, representation has the same vectors for the same concepts, visual concept and textual concepts.

33. 7:30

    That's a very important, um, thing to understand. And as you can see here,

34. 7:41

    it displays, um, the answer. Uh, I mean, the, the... I, I ask what's on the menu today. So it's displaying the, um, entries of the menu in French with the English, uh, translation because I asked the question in French.

35. 7:57

    Uh, and I could also... Oh, yeah, I asked-- Okay, that's funny. Um, because I asked what's good on the menu today. And so

36. 8:07

    the choice of what's good would depend on your personal taste preferences. [laughs]

37. 8:12

    Uh, and, uh, the menu offers a variety of traditional French dishes that could cater anyway, um, today. So yeah. So let's move on now to the next, uh, demo.

38. 8:24

    So we looked at, uh, you know, something that you might wanna do on the-- at the restaurant, uh, when you are in a foreign country and you don't understand what's in the menu, and you want to u- to get a better understanding, uh, if you have a, a special diet.

39. 8:37

    So that's very, uh, convenient. But that technology can also be used for more serious, um, challenges, uh, or use cases. So in this case,

40. 8:50

    we're going to look... And that's actually, uh, an actual use case, um, fr- from a discussion I had a couple of weeks ago with a customer working in the energy industry.

41. 9:01

    Um, and so here we have, um, a picture of electric poles, uh, that fell on the ground. And I can ask a very open question. What's going on here?

42. 9:20

    By the way, you can see how fast the model replies, which is quite something.

43. 9:27

    So not only is GPT-4o, um, understanding both images and text, but it's also much faster, uh, at, uh, answering. So as you can see here, the image shows several power lines and utility poles that have fallen or are leaning, indicating damage to the infrastructure.

44. 9:45

    I'm not gonna read everything, but what matters here is if I was, uh, working in the energy transport industry, uh, I might want to observe, um, continuously all the infrastructure of all the networks, like for a whole country, um, uh, at the edge to make sure that the network is operational.

45. 10:07

    So I might want to automate looking at all the video cameras, uh, of, uh, filming the, um, the, the infrastructure. So I could ask, "Is the electricity working here?"

46. 10:28

    It is highly unlikely that the electricity is working in the area shown in the image. Of course, here I ask the question in natural language, and the answer is presented to me in natural language too.

47. 10:40

    But I could also ask for the output to be generated in JSON in a format that could be interpreted by code so that I can automate, um, like dashboards and monitoring of, uh, re- infrastructure in real time.

48. 10:58

    Um, another use case is for the insurance industry. So here we have a house that we can ask what happened.

49. 11:13

    The image shows a house that has collapsed and is severely ti- tilted. Uh, uh, natural disasters such as hurricanes, earthquakes, or landslide. Um, so yeah, that's also a very interesting use case for the insurance industry.

50. 11:36

    Next. So, uh, so next time-- The next one is not gonna be a surprise because there was a spoiler. But, um, so we talked about, um,

51. 11:46

    multimodal models, but now we're gonna talk about another modality, uh, speech. Uh, the, um, Azure AI, uh, team, uh, product team has released, uh, an amazing new feature which allows you to translate, uh, videos.

52. 12:04

    So here I'm gonna play that video. So disclaimer, that's me on the video. [laughs]

53. 12:11

    Um, this is our new video translation service. With this, I can translate videos into other languages in my own voice. Jetzt kann ich Deutsch sprechen, wie ich es immer wollte.

54. 12:24

    Me hubiera gustado saber hablar español, pero ahora puedo hablarlo sin haber aprendido el idioma. Posso anche sussurrare in italiano. そして日本語で大きな声で話してください。 This will make the world more inclusive.

55. 12:39

    So what's really impressive about, um, that video is not only the fact that now I can speak German. [laughs] But [laughs], um, it took into consideration, uh, the intonation of what I was saying.

56. 12:54

    So when I was whispering, it was whispering, too. When I was yelling, it was yelling, too. Um, so it, it takes into account, um, the, the language and, and the tone.

57. 13:07

    A disclaimer, I stitched the different videos together myself using post-processing. But apart from that, I didn't do anything. Like, the, the, the service did that all by itself.

58. 13:19

    Um, now... Yes? Um, so as I understand, these are, uh, these are, uh, native models. Uh, do these success embeddings models as well? Uh. What is success embeddings? Okay, I'm gonna talk about embeddings models in a minute.

59. 13:31

    Okay. [laughs] Um, so, tak, tak, tak, where was I? Uh, model catalog. And thank you. That's a good segue, actually, because... So like I was saying, like, a, a, a year ago when GPT-4, ChatGPT was released, uh, basically you had almost no choice.

60. 13:51

    Now, the amount of models available on the Azure AI model catalog is extraordinary. Um, so here, uh, I'm gonna remove, uh, that, uh, filter here so that we display all of them.

61. 14:07

    So as you can see here, we have 1,600 models available in the model catalog right now. And what I like, there is one specific feature that I really like, is deployment options.

62. 14:23

    Here you can select serverless API. So you have two ways to de- to deploy, uh, models on Azure AI at the moment. You can deploy them serverless or you can, uh, deploy them, uh, using your infrastructure.

63. 14:39

    Uh, bring your own infrastructure means basically that you, uh, rent for GPUs, that you pay for GPUs whether you use the endpoint or not. Uh, serverless means that you pay for the token and that the infrastructure is, uh, managed for you by the vendor.

64. 14:57

    Um, and paying by the token is nothing new. You've been doing that with OpenAI GPT, uh, ever since it was released. But now you can do it for many vendors on the marketplace.

65. 15:08

    And as you can see here, those are all the vendors and all the models that are available on the catalog right now serverless. So you pay by the token and you have n- nothing to, um, manage yourself.

66. 15:20

    Uh, not to mention the fact that, um, getting GPUs right now is not the easiest. Uh, so being able to use those models serverless is... makes it much easier.

67. 15:31

    Um, and because we have so many models now, it- it's kind of hard to know which one to use. So now we also have the model benchmarks where we compare, not all, but we compare many of the models that are available in the catalog.

68. 15:46

    And you can look at the accuracy as, as, as well as a bunch of other metrics to figure out which model you wanna use for your use case.

69. 15:58

    One... Okay, let's hope that the Wi-Fi is not dead. Okay. Uh, one of those models, uh, w- that I want to focus on today, because we talked about GPT-4o, which is a very big model, uh, able to do text and, uh, visual analysis.

70. 16:18

    But, and Microsoft Research also came up with our own text and visual multimod- uh, modal model called Phi-3 Vision 128k. Um, this model is very, very interesting.

71. 16:34

    It's a family of model. We have a vision version. We have, uh, many sizes. That one specifically is very interesting because, um, I'm going to upload one of the use case.

72. 16:47

    For example, randomly, the same use case as the one I, uh, talked about before with GPT-4o. Um, so is electricity, uh, electricity working here?

73. 17:05

    And so what's really interesting here, um, is that the model, um, being, despite being much smaller and the explanation is a bit simpler.

74. 17:22

    But still, Phi-3 Vision is able to analyze the image and give a very good answer about the fact that the electricity most likely is not working here because of an outage, uh, or disruption.

75. 17:35

    So now you have the, uh, possibility in addition to GPT-4o to use Phi-3 Vision for that.

76. 17:47

    And not only can you use Phi-3 Vision as a service on Azure AI, we also have in the Phi-3 family a very, very small model, uh, of 3.8, uh, billion parameters, which, uh, weighs under, uh, roughly two gigabytes.

77. 18:06

    Uh, and here, if I refresh, um, the, uh, window here, and Dan, I hope I didn't make a mistake. No, it's okay. And so model size two gigabytes. And here, Phi-3 3.8 gigabytes quantitized with four bits is run- is downloading in the browser and running locally on the edge using WebGPU,

78. 18:31

    which is a new HTML5, uh, specification allowing application and browser to get access to the GPU. And so here I can ask, um,

79. 18:41

    "What do you know about the Rivian R2?" And you, you know, you're... This is gonna be a segue to one of the next subjects I'm gonna be talking about after.

80. 19:01

    And so as you can see, you get an answer generated in the browser at... I mean, look at how fast it is for a model running locally. Um, so okay, uh, I have a Mac which has an Apple silicon and, uh, some kind of GPU support, but it's by no mean a horse- a horsepower, uh, machine.

81. 19:24

    Uh, it's a MacBook Pro M2, I believe. Uh, and as you can see, it's running very, very well in the browser. That means that I can disconnect the Wi-Fi.

82. 19:34

    So I'm not gonna do it now, but you could disconnect the Wi-Fi and, and it would still run locally. It's, um, which is also very interesting for, um, um,

83. 19:46

    you know, uh, to process, uh, sensitive data.

84. 19:50

    Next thing, chat. So here, w- if I ask the question, uh, "What are the, uh, different

85. 20:02

    Rivian models?" So I'm asking GPT-4 Turbo, um, which was last updated in April 2023. It knows about the Rivian R1T, the R1S.

86. 20:21

    Also, uh, Amazon's, um, Rivian truck, and that's all. Now

87. 20:29

    I can select an index to do RAG, Retrieval Augmented Generation, where I can ground the, my model into my own documents. And I'm not gonna show that now, but what I did, uh, befo- to prepare the demo, I just took a bunch of Wikipedia pages of Rivian models that I uploaded to the index,

88. 20:54

    um, and now I am querying. So if I ask again the same question, "What are the different

89. 21:04

    Rivian models?" Huh, is it bugging? [laughs]

90. 21:20

    That's the problem with live demos. Uh, let me clear. I'm gonna copy that.

91. 21:28

    I'm gonna copy, clear, and rerun. Okay. Uh, I'm gonna refresh.

92. 21:46

    Okay. I have the index still selected. Now I should be able to... Huh.

93. 21:53

    Um, is it the model? Let's try with Phi-3 Turbo.

94. 22:02

    Yeah. So now, um, so apparently we have a, a bug with the other model, but, uh, with Phi-3 Turbo it works. And here you can see that we have the R1T, the R1S, the newly released R3, and we should have...

95. 22:17

    It doesn't mention the R2, but um, but it me- it does mention the R3. Um, so if I ask, "What about the R2?"

96. 22:31

    Yeah. So it does know about it. Um, so this is very interesting because it allows to use an LLM on up-to-date, uh, information. Now, let's move on to the next demo.

97. 22:45

    Um, so I'm gonna skip that one, but what I can show here is one thing which is very important is evaluation of models. Because when you build a LLM, uh, application, you want to be able to evaluate how good they are.

98. 23:01

    And when you make modifications to the system prompt or you change your models on your application, you wanna make sure that it continues to work as expected. So inside Azure AI Studio, you have a feature called Evaluation, which allows you to run, um, a bunch of metrics.

99. 23:17

    Here I show coherence, groundedness, and relevance. Groundedness is something that is very important for RAG applications because you wanna make sure that the answer is grounded into the documents.

100. 23:28

     So this system allows you to do that monitor- moderately, uh, easily. And as you can see here, so I used a very simple dataset that I prepared for the demo.

101. 23:38

     So I have only one entry in my, um, evaluation dataset. But still, it shows, um, that for that question, which is, "What are the different Rivian models?" The coherence was four.

102. 23:50

     It was w- well grounded because it's evaluated between one and five.

103. 23:56

     Um, next, something very cool that I absolutely want to show you before we run out of time. So here, um, this is one example of how to build an agent with code interpreter.

104. 24:09

     Last Sunday, and that's real data from last Sunday, I went kite surfing, uh, in the [REDACTED:location], in the [REDACTED:location]. And I did a pretty good in a session that I recording using my watch.

105. 24:21

     And I exported the GPX file from, uh, my session from my watch. And now I can ask question, which I up- up- uploaded the file to Code Interpreter. And now I can ask question about it.

106. 24:35

     So I can say, "Hey, how long was I on the water?"

107. 24:46

     And so here, the LLM is going to automatically generate Python code and execute it in a sandbox to analyze the file that I uploaded, which is a GPX XML file.

108. 24:59

     So it's not a CSV. Usually, when you see those demos, they use CSVs, right? But here, I'm using an XML file, which is more complicated. Um, and you're gonna see that it should work.

109. 25:12

     Yeah, 42 minutes. That's roughly how much time I was on the water. And now I can ask,

110. 25:20

     "How many turns, how many tacks did I do?"

111. 25:27

     So a tack in, uh, sailing is basically a turn. Um, and here, this is a much harder question because asking how many tacks I did requires not only analyzing the GPS coordinates, but also analyzing the d- the angular differentiation, difference between each point and applying a threshold to decide which of the points on the

112. 25:52

     path are actually turns. And as you can see here, it replies with 217. And now we can ask,

113. 26:01

     "Can you draw my session on a map?"

114. 26:12

     Because I wanna see visually what it looks like, right? It's more fun. Um, so while it generate... What's very interesting here is that I have no expertise in GPX.

115. 26:23

     Like, I don't know the file format. I could, uh... Like, if I show you what it looks like, it's pretty, um, uh, you know, technical and, uh, it's very hard to parse.

116. 26:33

     So here, without any explanation of what the file format is, Code Interpreter figured that out by itself. And here,

117. 26:48

     here's the map that it drew. And I'm gonna skip. I could have asked, "Hey, can you please..." And I did that to prepare the demo. "Can you please add draw red crosses for each one of my turns?"

118. 27:02

     And here is the result. Can you imagine how powerful that is? Like, I didn't code a single line.

119. 27:12

     Um, and 57. Can I show last thing? [laughs]

120. 27:22

     Uh, well, re- real quick. Um, here's, uh, something pretty cool that I wanna show you too. Uh, this is GitHub Workspaces, and here I can go to that repository, and I can...

121. 27:36

     So it's a preview, pre-access, and I can ask, "Can you add a Java GUI

122. 27:48

     frontend?" So this is a demo. Uh, I don't know if you noticed, but that was Python code. And I ask, "Can you generate a Java GUI frontend?" And it's gonna automatically, using an LLM, figure out what is the state of the r- code repository, figure out the, uh, what it contains, what it does not.

123. 28:10

     So it's gonna, um, write specifications, and based on the specification, we can ask to generate a plan. And every step of the way, if it makes a mistake, we can ask it to make corrections.

124. 28:24

     This is preview. This is not yet available. Uh, but this is incredibly powerful. Uh, and that's upcoming. [claps hands]

125. 28:34

     And I'm over. [laughs] [upbeat music]