AI Engineer World's Fair 2024
Multi-model, multimodal, and multi-agent innovations in Azure AI
Read the talk
From menu photos to executable analysis with Azure AI
What is vegan on a photographed menu? That small question opens a progression through multimodal reasoning, local inference, retrieval grounding and agents that execute Python.
From a talk by Cedric Vidal
Before you start: Basic familiarity with language-model prompts and APIs is helpful; the Python example assumes you can read a short script.
Beyond a text-only application
What changes when an AI application can work across models, images, speech and executable tools? In this AI Engineer World’s Fair 2024 walkthrough, Cédric Vidal, Microsoft’s principal AI advocate, explores those capabilities through demonstrations. Azure AI Studio provides the common environment, bringing together AI services, machine learning and responsible AI tooling. Its model offerings extend beyond OpenAI to other providers of text, vision and speech capabilities.
The interface and availability statements belong to that 2024 setting. Vidal introduces the Build announcements: Azure AI Studio’s general availability; GPT-4o, with text and vision available and speech described as forthcoming; Microsoft Research’s Phi-3 family; GPT-4 Turbo with Vision; DALL·E 3; and Whisper. The development features include the Assistants API, GPT-4 fine-tuning, batch inference and video translation. These establish the toolkit for the demonstrations, rather than a current availability guide.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From image descriptions to monitoring signals
A discussion with an energy-industry customer motivates the next example: a photograph of fallen utility poles and power lines. Asked what is happening, GPT-4o describes poles that have fallen or are leaning and identifies infrastructure damage. Vidal remarks on the quick response, but supplies no latency measurement. The useful change from the restaurant example is the question’s operational purpose: a description could help someone assess an electricity network.
Vidal then proposes continuous inspection of camera feeds across a network, potentially at national scale and at the edge. Asking whether electricity is working produces an assessment that service is highly unlikely in the pictured area. This is an inference from visible damage, not an electrical measurement. He suggests requesting JSON output instead of prose so software could consume the assessment in dashboards and monitoring workflows; the continuous monitoring system itself is not demonstrated.
The insurance example follows the same pattern with a different image. A house appears collapsed and severely tilted. The response describes that damage and offers hurricanes, earthquakes or landslides as possible causes. Visible condition and possible explanation remain distinct: the photograph supports damage assessment more directly than it establishes which event caused the damage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Translating speech without flattening delivery
The next modality is speech. Vidal plays a video of himself transformed by Azure AI’s video translation service. The sequence moves through German and Spanish, then whispered Italian and loudly delivered Japanese. The demonstration is about more than changing the words: the translated speech retains his voice and follows the expressive delivery of the original.
Vidal highlights that whispering stays quiet and yelling stays loud. He also separates the service’s work from his own editing: he manually stitched the separately translated videos into the multilingual sequence, while attributing the voice and language transformations to the service.
An audience question asks about embedding models. Vidal says he will return to them, then moves into the model catalog; the walkthrough does not return to an explicit embeddings explanation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing a model also means choosing how to run it
Opening the catalog and removing a filter makes the expanded model choice tangible. Vidal reports 1,600 models in the Azure AI catalog at recording time. That is the whole catalog, not the number available through serverless APIs. The deployment-options filter narrows the selection to models offered in that form.
| Deployment option | What you pay for | Infrastructure responsibility |
|---|---|---|
| Rented GPU infrastructure | GPU capacity, including idle time | You provision the capacity |
| Serverless API | Tokens consumed | Vendor manages infrastructure |
The serverless option extends a familiar OpenAI-style consumption model to additional marketplace vendors. Vidal also points to GPU scarcity: avoiding the need to obtain and manage capacity can be as useful as avoiding idle charges.
More choice creates a selection problem. Azure AI Studio’s model benchmarks compare many—but not all—catalog models using accuracy and other metrics. These comparisons help narrow candidates for a use case; the next demonstration explores whether a smaller model can handle a task already shown with GPT-4o.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeat the infrastructure question with Phi-3 Vision
Microsoft Research’s Phi-3 Vision 128k provides another text-and-image model within the Phi-3 family. Vidal uploads the same damaged-infrastructure photograph and asks the same question about whether electricity is working. Keeping the image and question fixed makes the difference in responses easier to inspect.
The smaller model gives a simpler explanation, but still identifies a likely outage or disruption. Vidal judges that answer useful for the task. This is a qualitative comparison of the two responses, with no comparative accuracy score: it shows a reason to evaluate the smaller model, rather than establishing that the models are interchangeable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move inference into the browser
The next Phi-3 demonstration changes the execution location. Vidal describes a 3.8-billion-parameter Phi-3 model, quantized to four bits, with an approximately 2 GB browser download. Parameters and download size are different quantities; this is a separate small-model example from the preceding vision demonstration. After downloading, the model runs locally through WebGPU, which gives browser applications access to the device’s GPU.
The prompt asks what the model knows about the Rivian R2. Text is generated inside the browser on Vidal’s Apple-silicon laptop, which he tentatively identifies as a MacBook Pro M2. He points out the apparent speed, without measuring throughput.
Local execution changes the network dependency and where sensitive inputs can be processed. Vidal says the downloaded model could continue running with Wi-Fi disconnected, although he leaves the connection on during the demonstration. The practical attraction is inference that can happen on the user’s device rather than requiring every input to be sent to a hosted model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the model newer Rivian information
The Rivian question carries into the next demonstration. Asked for the different Rivian models, GPT-4 Turbo, described here as last updated in April 2023, lists the R1T, R1S and Amazon delivery truck. The answer illustrates a knowledge-freshness problem: the application needs information beyond what this deployment supplies unaided.
Vidal selects a prebuilt index containing Wikipedia pages about Rivian models. This enables retrieval-augmented generation, or RAG: documents become grounding material for the answer without retraining the model. The pages were uploaded before the session; the live operation is selecting the index and asking the same question again.
The query stalls. Vidal clears the conversation, reruns it and refreshes the page, with the index still selected. He then switches models, calling the replacement “Phi-3 Turbo,” and the request succeeds. He attributes the earlier failure to an apparent bug with the other model.
The grounded answer includes R1T, R1S and the newly announced R3, but omits R2. A follow-up asking specifically about R2 produces information about it. The sequence shows both the benefit and the remaining application problem: retrieval can make newer information available, while an initial answer may still leave out a relevant item.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check behavior after changing prompts or models
Switching models raises the next engineering question: does the application still behave as intended? Vidal opens Azure AI Studio’s Evaluation feature, which exposes coherence, groundedness and relevance. The same need arises after editing a system prompt. For RAG, groundedness is especially useful because it assesses whether the answer is supported by the supplied documents.
For the single evaluation question about Rivian models, Vidal reports a coherence score of 4 on a 1–5 scale. He also describes the answer as well grounded, without clearly supplying a separate groundedness score. With only one dataset entry, this demonstrates how to inspect an evaluation result, not broad coverage of application behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask questions of a kitesurfing GPS track
The agent demonstration begins with a personal file: a kitesurfing session in the Bay, recorded on Vidal’s watch the preceding Sunday and exported as GPX. After uploading it to Code Interpreter, he asks how long he was on the water. The agent’s job is to turn that natural-language request into an operation over the file.
Code Interpreter generates Python and executes it in a sandbox. Here the input is XML-based GPX, rather than the CSV commonly used in data-analysis demonstrations. For the duration question, the central operation is extracting track-point timestamps and calculating the elapsed interval. A compact Python implementation of that operation is:
python
from datetime import datetime
from pathlib import Path
import xml.etree.ElementTree as ET
def track_duration_minutes(path: Path) -> float:
root = ET.parse(path).getroot()
timestamps = []
for point in root.findall(".//{*}trkpt"):
text = point.findtext("{*}time")
if text:
timestamps.append(
datetime.fromisoformat(text.strip().replace("Z", "+00:00"))
)
if len(timestamps) < 2:
raise ValueError("The track needs at least two timestamped points.")
elapsed = max(timestamps) - min(timestamps)
return elapsed.total_seconds() / 60
if __name__ == "__main__":
minutes = track_duration_minutes(Path("session.gpx"))
print(f"Recorded track span: {minutes:.1f} minutes")
This computes the recorded track’s elapsed span; whether that exactly matches time on the water depends on when recording began and ended. In the demonstration, Code Interpreter returns approximately 42 minutes, which Vidal says matches his recollection.
He then asks how many turns, or tacks, he made. This is a more interpretive operation than subtracting timestamps:
- Read the GPS coordinates along the track.
- Determine the direction of successive movements.
- Calculate angular changes along the path.
- Apply a threshold to classify changes as turns.
The agent reports 217 turns, without an independent count or a stated threshold. The threshold matters because the procedure must distinguish a meaningful maneuver from smaller changes in the recorded path.
Next, Vidal asks the agent to draw the session on a map. While it works, he emphasizes that he did not explain GPX’s structure or write the parsing code himself. The resulting map exposes the track visually. He then shows a prepared follow-up result with red crosses marking turns; that additional generation is skipped live. The progression moves from a scalar answer, to a derived count, to a plot that makes the analysis inspectable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a repository request to an editable plan
The final demonstration moves from interpreting a data file to understanding a code repository. Vidal opens a preview he calls GitHub Workspaces; the interface is labeled Copilot Workspace. The repository contains Python code, and the request is to add a Java GUI frontend. Before proposing changes, the system examines what the repository currently contains and which requested capabilities are missing.
The workflow first writes a specification, separating the current state from the proposed state. From that specification, the user can request a plan. Corrections are possible at each step, so the feature request becomes an inspectable sequence of decisions rather than an opaque jump to a code change.
Vidal presents this as a preview, not a generally available product at the time. The demonstrated endpoint is specification and planning, not a completed Java frontend or a deployment. Where Code Interpreter executed a bounded analysis over an uploaded file, this last workflow makes the proposed repository changes available for correction before implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Model specifications, weights and inference examples for Microsoft's Phi-3 text-and-image model.
Further reading
Microsoft's contemporary overview of the model catalog, GPT-4o, Phi-3 and preview video translation.
The original report on Phi-3-mini, its training approach and small-model capabilities.
- Rivian introduces R2 and R3Article
The March 2024 vehicle announcement supplies historical context for the talk's retrieval-grounding example.
A practical Microsoft tutorial for inspecting and running coherence, groundedness, relevance and fluency evaluation prompts.
Read the complete timestamped transcript
- 0:00
[on hold music] Uh, so, um, I'm Cédric Vidal.
- 0:16
I'm an, uh, principal AI advocate at Microsoft. Uh, and today, uh, I'm going to, uh... We're going to do quite cool stuff. Uh, we're going to do-- to talk about multi many things.
- 0:29
Uh, multi, uh, models, multimodality, multilingual, multi-agents. Uh, all of this with, uh, Azure AI. Um, so yeah. And also one particularity is, uh, apart from a few slides at the beginning, it's only demos.
- 0:45
Uh, and to be honest, bear with me in case, uh, one of them or all of them don't work. [laughs] Uh, but, uh, it's gonna be fun. We'll see how it goes.
- 0:55
Um, so as you know, you know, Azure AI is, uh, the best AI platform, uh, out there. Uh, we have, um, a lot of, uh, AI, uh, services. Uh, we can do, uh, machine learning.
- 1:09
Uh, and we also, uh, do all of that responsibly with the whole responsible AI framework. And we encapsulate all of this, um, in the Azure AI Studio, and I'm going to do, uh, a lot of demos, uh, of i-- Azure AI Studio today.
- 1:28
And, um, since now almost a year, a bit more than a year, um, we've been, uh, partnering with OpenAI, of course. Uh, and we have all of the OpenAI models available on Azure on the Azure platform.
- 1:41
But, uh, we're gonna see that, uh, in addition to all the Mo-- OpenAI models that we have and all the modalities that we can get using those, we also have many more models available o-on the platform for, um, uh, text of course, uh, vision, uh, and speech.
- 2:01
Um, and many organizations trust us today, uh, to, uh, use, uh, AI and build their products. Um,
- 2:11
so without, um, further ado. So, uh, I'm going to jump in the demos very quickly. But before I do, so we've had many things announced at Build a couple of months ago.
- 2:24
Uh, we've had, uh, the GA version of Azure AI Studio. We've had, uh, the latest, uh, model from OpenAI, GPT-4, uh, Omni, uh, which supports, uh, text, vision, uh, and soon speech.
- 2:37
Uh, we've had the new small la-language models, um, from Microsoft, from MS Research called Phi-3. Um, now we have also announced GPT-4 Turbo with Vision, uh, DALL·E 3, and Whisper.
- 2:51
Um, we've announced the Assistants, uh, API that allows to, uh, build, uh, agents, and I'm going to demo it. Uh, we've announced the fine-tuning for GPT-4, the new, uh, inference batch API.
- 3:06
Um, and um, also another very cool thing I'm going to demo today, and you had a glimpse of it. Uh, I mean, I guess the surprise is kind of, um, uh, out, uh, but the video, video translation service that I'm going to demo.
- 3:21
Um, and Azure AI Studio. Oh, yeah. So let's go straight to the demos now. Um, so apart from those slides, now it's only demos.
- 3:32
So the fun begins. Um, okay. The first demo. So we've had, um, Azure. I mean, uh, uh, like a year ago when, uh, everything started, uh, you didn't have much, uh, many choices.
- 3:48
It was basically, uh, GPT or, or GPT. The, the only modality available was text. Um, but now things have changed dramatically. Now, we support also, uh, multimodal vision, mixing text and vision, and this opens a completely new era of use cases.
- 4:07
For example, here and, um, let me zoom. So I'm going to demo, uh, GPT-4, um, o. And actually I selected Vision here, but what I wanted to select was GPT-4o.
- 4:20
Um, and I'm going to demonstrate a use case where... So it is kind of small right now, but, uh, this is, um, a menu from a restaurant. Um, and I'm going to ask what's vegan on the menu today.
- 4:38
So here we can see the menu a bit better.
- 4:42
So as you can see, we have, uh, winter chicory salad, uh, duck, sea bass, et cetera. Um, and what's very interesting here is that, uh, the menu that you just saw was, um, the, the font is funny, but it's printed.
- 4:58
Uh, so it's a font from a computer. Uh, and GPT-4o does a very good job at reading what's on the menu. Um, and let me zoom here so that we can see what's...
- 5:11
So I asked whether there were vegan options today on the menu. And what's interesting is that it looks, it mixes vision, so it extracted all the text from the image.
- 5:23
But not, not only that, but it reasons on it. So it analyzed all the items on the menu today and for each one of them, uh, looked at which ones were, uh, vegan.
- 5:33
And as you can see here, uh, the cauliflower soup and winter, um, uh, chicory salad. Uh, let me zoom. Ah.
- 5:44
Up. Okay. So, okay. Both mention, uh, vege-vegetarian and vegan versions. Um, so that's a very good example of how to mix, uh, text and reasoning.
- 6:00
Something that was not possible before with just OCRs, which becomes available with the new generation of multimodal, um, models.
- 6:10
Another example, slightly harder because this one has
- 6:20
handwritten text. So this menu has not been printed. It has been written by hand on a chalkboard.
- 6:33
So as you can see... Oh, and it's in French. Um, so not only does it recognize handwritten, um, uh, sentences written on a chalkboard in a picture, but it also translates it and reasons on it.
- 6:54
So that's three things that the model is doing all at once, thanks to, uh, multimodality. Uh, this is very important to understand how it differs from what we were doing before, because before we were using image to text to extract the text and then reason on it.
- 7:12
Now, the model understands natively both pixels and text, and in its internal, um, representation has the same vectors for the same concepts, visual concept and textual concepts.
- 7:30
That's a very important, um, thing to understand. And as you can see here,
- 7:41
it displays, um, the answer. Uh, I mean, the, the... I, I ask what's on the menu today. So it's displaying the, um, entries of the menu in French with the English, uh, translation because I asked the question in French.
- 7:57
Uh, and I could also... Oh, yeah, I asked-- Okay, that's funny. Um, because I asked what's good on the menu today. And so
- 8:07
the choice of what's good would depend on your personal taste preferences. [laughs]
- 8:12
Uh, and, uh, the menu offers a variety of traditional French dishes that could cater anyway, um, today. So yeah. So let's move on now to the next, uh, demo.
- 8:24
So we looked at, uh, you know, something that you might wanna do on the-- at the restaurant, uh, when you are in a foreign country and you don't understand what's in the menu, and you want to u- to get a better understanding, uh, if you have a, a special diet.
- 8:37
So that's very, uh, convenient. But that technology can also be used for more serious, um, challenges, uh, or use cases. So in this case,
- 8:50
we're going to look... And that's actually, uh, an actual use case, um, fr- from a discussion I had a couple of weeks ago with a customer working in the energy industry.
- 9:01
Um, and so here we have, um, a picture of electric poles, uh, that fell on the ground. And I can ask a very open question. What's going on here?
- 9:20
By the way, you can see how fast the model replies, which is quite something.
- 9:27
So not only is GPT-4o, um, understanding both images and text, but it's also much faster, uh, at, uh, answering. So as you can see here, the image shows several power lines and utility poles that have fallen or are leaning, indicating damage to the infrastructure.
- 9:45
I'm not gonna read everything, but what matters here is if I was, uh, working in the energy transport industry, uh, I might want to observe, um, continuously all the infrastructure of all the networks, like for a whole country, um, uh, at the edge to make sure that the network is operational.
- 10:07
So I might want to automate looking at all the video cameras, uh, of, uh, filming the, um, the, the infrastructure. So I could ask, "Is the electricity working here?"
- 10:28
It is highly unlikely that the electricity is working in the area shown in the image. Of course, here I ask the question in natural language, and the answer is presented to me in natural language too.
- 10:40
But I could also ask for the output to be generated in JSON in a format that could be interpreted by code so that I can automate, um, like dashboards and monitoring of, uh, re- infrastructure in real time.
- 10:58
Um, another use case is for the insurance industry. So here we have a house that we can ask what happened.
- 11:13
The image shows a house that has collapsed and is severely ti- tilted. Uh, uh, natural disasters such as hurricanes, earthquakes, or landslide. Um, so yeah, that's also a very interesting use case for the insurance industry.
- 11:36
Next. So, uh, so next time-- The next one is not gonna be a surprise because there was a spoiler. But, um, so we talked about, um,
- 11:46
multimodal models, but now we're gonna talk about another modality, uh, speech. Uh, the, um, Azure AI, uh, team, uh, product team has released, uh, an amazing new feature which allows you to translate, uh, videos.
- 12:04
So here I'm gonna play that video. So disclaimer, that's me on the video. [laughs]
- 12:11
Um, this is our new video translation service. With this, I can translate videos into other languages in my own voice. Jetzt kann ich Deutsch sprechen, wie ich es immer wollte.
- 12:24
Me hubiera gustado saber hablar español, pero ahora puedo hablarlo sin haber aprendido el idioma. Posso anche sussurrare in italiano. そして日本語で大きな声で話してください。 This will make the world more inclusive.
- 12:39
So what's really impressive about, um, that video is not only the fact that now I can speak German. [laughs] But [laughs], um, it took into consideration, uh, the intonation of what I was saying.
- 12:54
So when I was whispering, it was whispering, too. When I was yelling, it was yelling, too. Um, so it, it takes into account, um, the, the language and, and the tone.
- 13:07
A disclaimer, I stitched the different videos together myself using post-processing. But apart from that, I didn't do anything. Like, the, the, the service did that all by itself.
- 13:19
Um, now... Yes? Um, so as I understand, these are, uh, these are, uh, native models. Uh, do these success embeddings models as well? Uh. What is success embeddings? Okay, I'm gonna talk about embeddings models in a minute.
- 13:31
Okay. [laughs] Um, so, tak, tak, tak, where was I? Uh, model catalog. And thank you. That's a good segue, actually, because... So like I was saying, like, a, a, a year ago when GPT-4, ChatGPT was released, uh, basically you had almost no choice.
- 13:51
Now, the amount of models available on the Azure AI model catalog is extraordinary. Um, so here, uh, I'm gonna remove, uh, that, uh, filter here so that we display all of them.
- 14:07
So as you can see here, we have 1,600 models available in the model catalog right now. And what I like, there is one specific feature that I really like, is deployment options.
- 14:23
Here you can select serverless API. So you have two ways to de- to deploy, uh, models on Azure AI at the moment. You can deploy them serverless or you can, uh, deploy them, uh, using your infrastructure.
- 14:39
Uh, bring your own infrastructure means basically that you, uh, rent for GPUs, that you pay for GPUs whether you use the endpoint or not. Uh, serverless means that you pay for the token and that the infrastructure is, uh, managed for you by the vendor.
- 14:57
Um, and paying by the token is nothing new. You've been doing that with OpenAI GPT, uh, ever since it was released. But now you can do it for many vendors on the marketplace.
- 15:08
And as you can see here, those are all the vendors and all the models that are available on the catalog right now serverless. So you pay by the token and you have n- nothing to, um, manage yourself.
- 15:20
Uh, not to mention the fact that, um, getting GPUs right now is not the easiest. Uh, so being able to use those models serverless is... makes it much easier.
- 15:31
Um, and because we have so many models now, it- it's kind of hard to know which one to use. So now we also have the model benchmarks where we compare, not all, but we compare many of the models that are available in the catalog.
- 15:46
And you can look at the accuracy as, as, as well as a bunch of other metrics to figure out which model you wanna use for your use case.
- 15:58
One... Okay, let's hope that the Wi-Fi is not dead. Okay. Uh, one of those models, uh, w- that I want to focus on today, because we talked about GPT-4o, which is a very big model, uh, able to do text and, uh, visual analysis.
- 16:18
But, and Microsoft Research also came up with our own text and visual multimod- uh, modal model called Phi-3 Vision 128k. Um, this model is very, very interesting.
- 16:34
It's a family of model. We have a vision version. We have, uh, many sizes. That one specifically is very interesting because, um, I'm going to upload one of the use case.
- 16:47
For example, randomly, the same use case as the one I, uh, talked about before with GPT-4o. Um, so is electricity, uh, electricity working here?
- 17:05
And so what's really interesting here, um, is that the model, um, being, despite being much smaller and the explanation is a bit simpler.
- 17:22
But still, Phi-3 Vision is able to analyze the image and give a very good answer about the fact that the electricity most likely is not working here because of an outage, uh, or disruption.
- 17:35
So now you have the, uh, possibility in addition to GPT-4o to use Phi-3 Vision for that.
- 17:47
And not only can you use Phi-3 Vision as a service on Azure AI, we also have in the Phi-3 family a very, very small model, uh, of 3.8, uh, billion parameters, which, uh, weighs under, uh, roughly two gigabytes.
- 18:06
Uh, and here, if I refresh, um, the, uh, window here, and Dan, I hope I didn't make a mistake. No, it's okay. And so model size two gigabytes. And here, Phi-3 3.8 gigabytes quantitized with four bits is run- is downloading in the browser and running locally on the edge using WebGPU,
- 18:31
which is a new HTML5, uh, specification allowing application and browser to get access to the GPU. And so here I can ask, um,
- 18:41
"What do you know about the Rivian R2?" And you, you know, you're... This is gonna be a segue to one of the next subjects I'm gonna be talking about after.
- 19:01
And so as you can see, you get an answer generated in the browser at... I mean, look at how fast it is for a model running locally. Um, so okay, uh, I have a Mac which has an Apple silicon and, uh, some kind of GPU support, but it's by no mean a horse- a horsepower, uh, machine.
- 19:24
Uh, it's a MacBook Pro M2, I believe. Uh, and as you can see, it's running very, very well in the browser. That means that I can disconnect the Wi-Fi.
- 19:34
So I'm not gonna do it now, but you could disconnect the Wi-Fi and, and it would still run locally. It's, um, which is also very interesting for, um, um,
- 19:46
you know, uh, to process, uh, sensitive data.
- 19:50
Next thing, chat. So here, w- if I ask the question, uh, "What are the, uh, different
- 20:02
Rivian models?" So I'm asking GPT-4 Turbo, um, which was last updated in April 2023. It knows about the Rivian R1T, the R1S.
- 20:21
Also, uh, Amazon's, um, Rivian truck, and that's all. Now
- 20:29
I can select an index to do RAG, Retrieval Augmented Generation, where I can ground the, my model into my own documents. And I'm not gonna show that now, but what I did, uh, befo- to prepare the demo, I just took a bunch of Wikipedia pages of Rivian models that I uploaded to the index,
- 20:54
um, and now I am querying. So if I ask again the same question, "What are the different
- 21:04
Rivian models?" Huh, is it bugging? [laughs]
- 21:20
That's the problem with live demos. Uh, let me clear. I'm gonna copy that.
- 21:28
I'm gonna copy, clear, and rerun. Okay. Uh, I'm gonna refresh.
- 21:46
Okay. I have the index still selected. Now I should be able to... Huh.
- 21:53
Um, is it the model? Let's try with Phi-3 Turbo.
- 22:02
Yeah. So now, um, so apparently we have a, a bug with the other model, but, uh, with Phi-3 Turbo it works. And here you can see that we have the R1T, the R1S, the newly released R3, and we should have...
- 22:17
It doesn't mention the R2, but um, but it me- it does mention the R3. Um, so if I ask, "What about the R2?"
- 22:31
Yeah. So it does know about it. Um, so this is very interesting because it allows to use an LLM on up-to-date, uh, information. Now, let's move on to the next demo.
- 22:45
Um, so I'm gonna skip that one, but what I can show here is one thing which is very important is evaluation of models. Because when you build a LLM, uh, application, you want to be able to evaluate how good they are.
- 23:01
And when you make modifications to the system prompt or you change your models on your application, you wanna make sure that it continues to work as expected. So inside Azure AI Studio, you have a feature called Evaluation, which allows you to run, um, a bunch of metrics.
- 23:17
Here I show coherence, groundedness, and relevance. Groundedness is something that is very important for RAG applications because you wanna make sure that the answer is grounded into the documents.
- 23:28
So this system allows you to do that monitor- moderately, uh, easily. And as you can see here, so I used a very simple dataset that I prepared for the demo.
- 23:38
So I have only one entry in my, um, evaluation dataset. But still, it shows, um, that for that question, which is, "What are the different Rivian models?" The coherence was four.
- 23:50
It was w- well grounded because it's evaluated between one and five.
- 23:56
Um, next, something very cool that I absolutely want to show you before we run out of time. So here, um, this is one example of how to build an agent with code interpreter.
- 24:09
Last Sunday, and that's real data from last Sunday, I went kite surfing, uh, in the [REDACTED:location], in the [REDACTED:location]. And I did a pretty good in a session that I recording using my watch.
- 24:21
And I exported the GPX file from, uh, my session from my watch. And now I can ask question, which I up- up- uploaded the file to Code Interpreter. And now I can ask question about it.
- 24:35
So I can say, "Hey, how long was I on the water?"
- 24:46
And so here, the LLM is going to automatically generate Python code and execute it in a sandbox to analyze the file that I uploaded, which is a GPX XML file.
- 24:59
So it's not a CSV. Usually, when you see those demos, they use CSVs, right? But here, I'm using an XML file, which is more complicated. Um, and you're gonna see that it should work.
- 25:12
Yeah, 42 minutes. That's roughly how much time I was on the water. And now I can ask,
- 25:20
"How many turns, how many tacks did I do?"
- 25:27
So a tack in, uh, sailing is basically a turn. Um, and here, this is a much harder question because asking how many tacks I did requires not only analyzing the GPS coordinates, but also analyzing the d- the angular differentiation, difference between each point and applying a threshold to decide which of the points on the
- 25:52
path are actually turns. And as you can see here, it replies with 217. And now we can ask,
- 26:01
"Can you draw my session on a map?"
- 26:12
Because I wanna see visually what it looks like, right? It's more fun. Um, so while it generate... What's very interesting here is that I have no expertise in GPX.
- 26:23
Like, I don't know the file format. I could, uh... Like, if I show you what it looks like, it's pretty, um, uh, you know, technical and, uh, it's very hard to parse.
- 26:33
So here, without any explanation of what the file format is, Code Interpreter figured that out by itself. And here,
- 26:48
here's the map that it drew. And I'm gonna skip. I could have asked, "Hey, can you please..." And I did that to prepare the demo. "Can you please add draw red crosses for each one of my turns?"
- 27:02
And here is the result. Can you imagine how powerful that is? Like, I didn't code a single line.
- 27:12
Um, and 57. Can I show last thing? [laughs]
- 27:22
Uh, well, re- real quick. Um, here's, uh, something pretty cool that I wanna show you too. Uh, this is GitHub Workspaces, and here I can go to that repository, and I can...
- 27:36
So it's a preview, pre-access, and I can ask, "Can you add a Java GUI
- 27:48
frontend?" So this is a demo. Uh, I don't know if you noticed, but that was Python code. And I ask, "Can you generate a Java GUI frontend?" And it's gonna automatically, using an LLM, figure out what is the state of the r- code repository, figure out the, uh, what it contains, what it does not.
- 28:10
So it's gonna, um, write specifications, and based on the specification, we can ask to generate a plan. And every step of the way, if it makes a mistake, we can ask it to make corrections.
- 28:24
This is preview. This is not yet available. Uh, but this is incredibly powerful. Uh, and that's upcoming. [claps hands]
- 28:34
And I'm over. [laughs] [upbeat music]