# Multi-model, multimodal, and multi-agent innovations in Azure AI

Cedric Vidal, Microsoft

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/YcOrV-TLNyQ-multi-model-multimodal-multi-agent-innovations) · [Video](https://www.youtube.com/watch?v=YcOrV-TLNyQ)

## Multi-model, multimodal, and multi-agent innovations in Azure AI

What is vegan on a photographed menu? That small question opens a progression through multimodal reasoning, local inference, retrieval grounding and agents that execute Python.

Basic familiarity with language-model prompts and APIs is helpful; the Python example assumes you can read a short script.

### Beyond a text-only application

[0:16](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=16s)

What changes when an AI application can work across models, images, speech and executable tools? In this [AI Engineer World’s Fair 2024 walkthrough](https://ai.engineer/worldsfair/2024/schedule), Cédric Vidal, Microsoft’s principal AI advocate, explores those capabilities through demonstrations. Azure AI Studio provides the common environment, bringing together AI services, machine learning and responsible AI tooling. Its model offerings extend beyond OpenAI to other providers of text, vision and speech capabilities.

![Azure AI overview with a foundation-model banner above three columns for Azure AI Services, Azure Machine Learning, and Responsible AI Tooling, with Azure AI Studio below.](https://ai.engineer/api/talk-essay-media/ac0919b54295c2e096740c51863b62f7d2fdda74ee6e45fb725466a037b5666d)

Azure AI services, machine learning, and responsible AI tooling within Azure AI Studio.

The interface and availability statements belong to that 2024 setting. Vidal introduces the [Build announcements](https://azure.microsoft.com/en-us/blog/from-code-to-production-new-ways-azure-helps-you-build-transformational-ai-experiences/): Azure AI Studio’s general availability; **GPT-4o**, with text and vision available and speech described as forthcoming; Microsoft Research’s **Phi-3** family; **GPT-4 Turbo with Vision**; **DALL·E 3**; and **Whisper**. The development features include the Assistants API, GPT-4 fine-tuning, batch inference and video translation. These establish the toolkit for the demonstrations, rather than a current availability guide.

### What is vegan on this menu?

[3:32](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=212s)

The first practical problem is a restaurant menu photographed rather than supplied as text. Vidal corrects the model selection to GPT-4o and asks what is vegan on the menu. The image uses a stylized printed font and includes winter chicory salad, duck and sea bass. Reading the words is only the first part of the task: the model must then inspect the dishes and their descriptions for dietary suitability.

The answer identifies cauliflower soup and winter chicory salad because their descriptions mention vegetarian and vegan versions. That distinction matters: the result identifies available versions, not a guarantee that every preparation of those dishes is vegan. **Text recognition supplies the evidence; reasoning turns it into an answer to the diner’s question.** Standalone OCR would extract the menu text but leave the dietary assessment to another component.

The next menu adds two difficulties: it is handwritten on a chalkboard, and it is in French. The model recognizes the writing, translates it and reasons over the contents in the same interaction. Vidal contrasts this with a pipeline that first converts an image into text and then passes that text to a reasoning stage. He describes corresponding visual and textual concepts as sharing internal vectors—a conceptual explanation of cross-modal understanding, rather than an established account of GPT-4o’s exact architecture.

Because the question is in English, the response pairs French dish names with English translations. But the wording asks what is *good*, not simply what is available. The response qualifies its recommendations: what tastes good depends on personal preferences. Together, the two menus show why the user’s question matters as much as recognition quality. A traveler may need translation, dietary filtering or recommendations; those are different operations over the same image.

### From image descriptions to monitoring signals

[8:37](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=517s)

A discussion with an energy-industry customer motivates the next example: a photograph of fallen utility poles and power lines. Asked what is happening, GPT-4o describes poles that have fallen or are leaning and identifies infrastructure damage. Vidal remarks on the quick response, but supplies no latency measurement. The useful change from the restaurant example is the question’s operational purpose: a description could help someone assess an electricity network.

Vidal then proposes continuous inspection of camera feeds across a network, potentially at national scale and at the edge. Asking whether electricity is working produces an assessment that service is highly unlikely in the pictured area. This is an inference from visible damage, not an electrical measurement. He suggests requesting **JSON output** instead of prose so software could consume the assessment in dashboards and monitoring workflows; the continuous monitoring system itself is not demonstrated.

The insurance example follows the same pattern with a different image. A house appears collapsed and severely tilted. The response describes that damage and offers hurricanes, earthquakes or landslides as possible causes. Visible condition and possible explanation remain distinct: the photograph supports damage assessment more directly than it establishes which event caused the damage.

### Translating speech without flattening delivery

[11:36](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=696s)

The next modality is speech. Vidal plays a video of himself transformed by Azure AI’s video translation service. The sequence moves through German and Spanish, then whispered Italian and loudly delivered Japanese. The demonstration is about more than changing the words: the translated speech retains his voice and follows the expressive delivery of the original.

Vidal highlights that whispering stays quiet and yelling stays loud. He also separates the service’s work from his own editing: he manually stitched the separately translated videos into the multilingual sequence, while attributing the voice and language transformations to the service.

An audience question asks about embedding models. Vidal says he will return to them, then moves into the model catalog; the walkthrough does not return to an explicit embeddings explanation.

### Choosing a model also means choosing how to run it

[13:31](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=811s)

Opening the catalog and removing a filter makes the expanded model choice tangible. Vidal reports 1,600 models in the Azure AI catalog at recording time. That is the whole catalog, not the number available through serverless APIs. The deployment-options filter narrows the selection to models offered in that form.

| Deployment option | What you pay for | Infrastructure responsibility |
| --- | --- | --- |
| Rented GPU infrastructure | GPU capacity, including idle time | You provision the capacity |
| Serverless API | Tokens consumed | Vendor manages infrastructure |

The serverless option extends a familiar OpenAI-style consumption model to additional marketplace vendors. Vidal also points to GPU scarcity: avoiding the need to obtain and manage capacity can be as useful as avoiding idle charges.

More choice creates a selection problem. Azure AI Studio’s model benchmarks compare many—but not all—catalog models using accuracy and other metrics. These comparisons help narrow candidates for a use case; the next demonstration explores whether a smaller model can handle a task already shown with GPT-4o.

### Repeat the infrastructure question with Phi-3 Vision

[16:18](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=978s)

Microsoft Research’s [**Phi-3 Vision 128k**](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) provides another text-and-image model within the Phi-3 family. Vidal uploads the same damaged-infrastructure photograph and asks the same question about whether electricity is working. Keeping the image and question fixed makes the difference in responses easier to inspect.

The smaller model gives a simpler explanation, but still identifies a likely outage or disruption. Vidal judges that answer useful for the task. This is a qualitative comparison of the two responses, with no comparative accuracy score: it shows a reason to evaluate the smaller model, rather than establishing that the models are interchangeable.

![Zoomed chat response beneath “is electricity working here?” says downed utility lines and a disrupted power pole suggest an outage or disruption; the Phi-3 vision model identifier appears below.](https://ai.engineer/api/talk-essay-media/5d7da85388d1ceaf8e085c279185baf04e68037e2cec1c0fe475888fd820c80a)

Phi-3 Vision identifies signs of a possible power outage or disruption.

### Move inference into the browser

[17:47](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1067s)

The next Phi-3 demonstration changes the execution location. Vidal describes a 3.8-billion-parameter Phi-3 model, quantized to four bits, with an approximately 2 GB browser download. Parameters and download size are different quantities; this is a separate small-model example from the preceding vision demonstration. After downloading, the model runs locally through **WebGPU**, which gives browser applications access to the device’s GPU.

The prompt asks what the model knows about the Rivian R2. Text is generated inside the browser on Vidal’s Apple-silicon laptop, which he tentatively identifies as a MacBook Pro M2. He points out the apparent speed, without measuring throughput.

Local execution changes the network dependency and where sensitive inputs can be processed. Vidal says the downloaded model could continue running with Wi-Fi disconnected, although he leaves the connection on during the demonstration. The practical attraction is inference that can happen on the user’s device rather than requiring every input to be sent to a hosted model.

### Give the model newer Rivian information

[19:50](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1190s)

The Rivian question carries into the next demonstration. Asked for the different Rivian models, **GPT-4 Turbo**, described here as last updated in April 2023, lists the R1T, R1S and Amazon delivery truck. The answer illustrates a knowledge-freshness problem: the application needs information beyond what this deployment supplies unaided.

Vidal selects a prebuilt index containing Wikipedia pages about Rivian models. This enables **retrieval-augmented generation**, or RAG: documents become grounding material for the answer without retraining the model. The pages were uploaded before the session; the live operation is selecting the index and asking the same question again.

The query stalls. Vidal clears the conversation, reruns it and refreshes the page, with the index still selected. He then switches models, calling the replacement “Phi-3 Turbo,” and the request succeeds. He attributes the earlier failure to an apparent bug with the other model.

The grounded answer includes R1T, R1S and the newly announced [R3](https://rivian.com/stories/meet-r2-and-r3), but omits R2. A follow-up asking specifically about R2 produces information about it. The sequence shows both the benefit and the remaining application problem: retrieval can make newer information available, while an initial answer may still leave out a relevant item.

### Check behavior after changing prompts or models

[22:45](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1365s)

Switching models raises the next engineering question: does the application still behave as intended? Vidal opens Azure AI Studio’s Evaluation feature, which exposes coherence, groundedness and relevance. The same need arises after editing a system prompt. For RAG, **groundedness** is especially useful because it assesses whether the answer is supported by the supplied documents.

For the single evaluation question about Rivian models, Vidal reports a coherence score of 4 on a 1–5 scale. He also describes the answer as well grounded, without clearly supplying a separate groundedness score. With only one dataset entry, this demonstrates how to inspect an evaluation result, not broad coverage of application behavior.

### Ask questions of a kitesurfing GPS track

[23:56](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1436s)

The agent demonstration begins with a personal file: a kitesurfing session, recorded on Vidal’s watch the preceding Sunday and exported as GPX. After uploading it to Code Interpreter, he asks how long he was on the water. The agent’s job is to turn that natural-language request into an operation over the file.

**Code Interpreter generates Python and executes it in a sandbox.** Here the input is XML-based GPX, rather than the CSV commonly used in data-analysis demonstrations. For the duration question, the central operation is extracting track-point timestamps and calculating the elapsed interval. A compact Python implementation of that operation is:

```python
from datetime import datetime
from pathlib import Path
import xml.etree.ElementTree as ET


def track_duration_minutes(path: Path) -> float:
    root = ET.parse(path).getroot()
    timestamps = []

    for point in root.findall(".//{*}trkpt"):
        text = point.findtext("{*}time")
        if text:
            timestamps.append(
                datetime.fromisoformat(text.strip().replace("Z", "+00:00"))
            )

    if len(timestamps) < 2:
        raise ValueError("The track needs at least two timestamped points.")

    elapsed = max(timestamps) - min(timestamps)
    return elapsed.total_seconds() / 60


if __name__ == "__main__":
    minutes = track_duration_minutes(Path("session.gpx"))
    print(f"Recorded track span: {minutes:.1f} minutes")
```

This computes the recorded track’s elapsed span; whether that exactly matches time on the water depends on when recording began and ended. In the demonstration, Code Interpreter returns approximately 42 minutes, which Vidal says matches his recollection.

He then asks how many turns, or tacks, he made. This is a more interpretive operation than subtracting timestamps:

1. Read the GPS coordinates along the track.
2. Determine the direction of successive movements.
3. Calculate angular changes along the path.
4. Apply a threshold to classify changes as turns.

The agent reports 217 turns, without an independent count or a stated threshold. The threshold matters because the procedure must distinguish a meaningful maneuver from smaller changes in the recorded path.

Next, Vidal asks the agent to draw the session on a map. While it works, he emphasizes that he did not explain GPX’s structure or write the parsing code himself. The resulting map exposes the track visually. He then shows a prepared follow-up result with red crosses marking turns; that additional generation is skipped live. The progression moves from a scalar answer, to a derived count, to a plot that makes the analysis inspectable.

### From a repository request to an editable plan

[27:22](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1642s)

The final demonstration moves from interpreting a data file to understanding a code repository. Vidal opens a preview he calls GitHub Workspaces; the interface is labeled **Copilot Workspace**. The repository contains Python code, and the request is to add a Java GUI frontend. Before proposing changes, the system examines what the repository currently contains and which requested capabilities are missing.

The workflow first writes a specification, separating the current state from the proposed state. From that specification, the user can request a plan. Corrections are possible at each step, so the feature request becomes an inspectable sequence of decisions rather than an opaque jump to a code change.

![Copilot Workspace shows the request “Can you add a Java GUI frontend?”, completed Current and Proposed specification lists, and a Generate plan button.](https://ai.engineer/api/talk-essay-media/5262a0a503aa9a167027591cb0a0c24c188888b6bbcb06e163a52c8bd849a3cf)

Copilot Workspace specifies a proposed Java GUI frontend for a Python repository.

Vidal presents this as a preview, not a generally available product at the time. The demonstrated endpoint is specification and planning, not a completed Java frontend or a deployment. Where Code Interpreter executed a bounded analysis over an uploaded file, this last workflow makes the proposed repository changes available for correction before implementation.

## Resources

- [Azure AI announcements at Build 2024](https://azure.microsoft.com/en-us/blog/from-code-to-production-new-ways-azure-helps-you-build-transformational-ai-experiences/) — Microsoft's contemporary overview of the model catalog, GPT-4o, Phi-3 and preview video translation.
- [Phi-3 Vision 128K model card](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) — Model specifications, weights and inference examples for Microsoft's Phi-3 text-and-image model.
- [Phi-3 technical report, original version](https://arxiv.org/html/2404.14219v1) — The original report on Phi-3-mini, its training approach and small-model capabilities.
- [Rivian introduces R2 and R3](https://rivian.com/stories/meet-r2-and-r3) — The March 2024 vehicle announcement supplies historical context for the talk's retrieval-grounding example.
- [Understand and run quality evaluators](https://microsoft.github.io/aitour-build-a-copilot-on-azure-ai/2-Workshop/03-Evaluate/02/) — A practical Microsoft tutorial for inspecting and running coherence, groundedness, relevance and fluency evaluation prompts.

## Slides

### Presenting sponsor: Microsoft

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=5s) [Microsoft logo beneath the heading Presenting Sponsor.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d3d51b7f4395ac7638521872389073b482489fa8b12ec0e10957e6f0474a758.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/c3f9231a885ca3b7fdeeaf3ba33d6931c29a996b6b9b79979a69e4ef4b8e8da6.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Azure AI overview

Azure AI places foundation models above three areas: AI Services, Machine Learning, and Responsible AI Tooling. Azure AI Studio spans them as a platform for developing and deploying custom copilots.

[0:54](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=54s) [Azure AI overview showing foundation models, AI Services, Machine Learning, Responsible AI Tooling and Azure AI Studio.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bc581317f1a1c9248edeed2c6cb7852bde42b6e7dba1664e479a645221715dd.webp)

Azure AI

Best-in-class AI foundation models

Azure AI Services
Pre-trained, turnkey solutions for intelligent applications

Azure Machine Learning
Full-lifecycle tools for designing and managing AI models

Responsible AI Tooling
Build and manage apps that are trustworthy by design

Azure AI Studio
A comprehensive platform to develop and deploy custom copilots

### OpenAI and Microsoft

OpenAI and Microsoft pair their missions above Azure OpenAI Service, which groups models by language, multimodal, fine-tuning, image, and transcription and translation uses.

[1:26](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=86s) [OpenAI and Microsoft missions above an Azure OpenAI Service overview of language, multimodal, fine-tuning, image and speech capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/93f4f032b4d00c9572e66851cf76fb5aecf1ca2a725b3b01508847077b784549.webp)

Best Friends

OpenAI
Ensure that artificial general intelligence (AGI) benefits humanity

Microsoft
Empower every person and organization on the planet to achieve more

Azure OpenAI Service

Language
GPT-4, GPT-3.5-Turbo

Multi-Modal
GPT-4o
GPT-4-Turbo with Vision

Fine Tuning
GPT-4
GPT-3.5-Turbo

Images
DALL·E 3

Transcription & Translation
Whisper

On Your Data GA
Azure AI Studio GA
Assistants API Preview
Pricing Innovation

### Organizations using Azure AI

Azure AI reports 53,000+ organizations using its services, illustrated by logos ranging from Schneider Electric and Providence to the NBA and McDonald’s.

[2:00](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=120s) [The slide states that more than 53,000 organizations use Azure AI and displays a grid of organization logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/fbba30dd52527ce726fa52af3ec81759ba1d2e78416f08ee6898d91ae44dde5c.webp)

53,000+ organizations
using Azure AI today

Schneider Electric
MODERN Requirements
ASOS
TIM
Moveworks
UNIVERSAL DESTINATIONS & EXPERIENCES
vodafone
Unity
Schroders

Providence
Inflection
ARTHUR D LITTLE
Swift
H&R BLOCK
IVECO
VOLKSWAGEN GROUP
NBA

freshworks
SWECO
AT&T
CallMiner
otto group
SymphonyAI
UiPath
ally
DEEPBRAIN AI

Le Monde
Manulife
John Hancock
CARMAX
SIEMENS
EY
pwc
Government of South Australia
Department for Education
Project Gutenberg

Laerdal
helping save lives
USA TODAY
KPMG
arvato BERTELSMANN
reddit
Lexmark
HEINEKEN
ERM
DocuSign
M&S
EST. 1884

elastic
NTT
kry
CHANEL
DATADOG
STRABAG
Beiersdorf
VOLVO
ZIMMER BIOMET

### New in Azure AI

Azure AI Studio, GPT-4o and Phi-3 are generally available; the Assistants API, GPT-4 fine-tuning and video translation are in preview. The speaker plans demos of the Assistants API and video translation.

[2:12](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=132s) [Azure AI announcements list generally available products and preview features, including Azure AI Studio, GPT-4o, Phi-3, the Assistants API and video translation.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b80d03533c7376dc7a0f8f42bc2300c33e67a0af82d49dcbb6a499c68f5aa2c.webp)

New in Azure AI

General availability
Azure AI Studio

General availability
GPT-4o

General availability
Phi-3 SLMs

General availability
GPT-4 Turbo with Vision, DALL·E 3, Whisper

Generally Available (Refresh)
On your data

Preview (Refresh)
Assistants API

Preview
Fine Tuning for GPT-4

Preview
Batch API

Preview
Configurable Content Filters for Dall-E and GPT-4-Turbo

Preview
Video Translation

### Azure AI Studio

Azure AI Studio is generally available for developing and deploying custom copilots, with model choice, an AI toolchain, responsible AI tools and enterprise-scale production.

[3:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=200s) [Azure AI Studio is generally available, with model choice, AI tools, responsible AI tools, and enterprise-scale production.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb0c87aecd41293485ab5fb507a6f4f13df53cc42f1413eb9a7866211248ef92.webp)

Azure AI Studio
Generally Available

A comprehensive platform to develop and deploy custom copilots

API and Model Choice
Complete AI Toolchain
Responsible AI Tools
Enterprise-grade Production at Scale

AI.Azure.com

### Menu close-up

The presenter zooms in on a restaurant menu before asking what is vegan. The winter chicory salad offers a vegan version on request; duck, sea bass, and porcini tagliatelle appear below it.

[4:40](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=280s) [A close-up of a restaurant menu shows winter chicory salad with a vegan version available upon request.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9c0832b8ea06b082d9e94d86bf493f514188dcd055ca4c548d2fcec5412553c.webp)

OR
• Winter Chicory Salad
Persimmon, brioche crouton, saffron vinaigrette
(Vegetarian & vegan version available upon request[illegible]

• Grimmaud’s Farms Duck Breast
Potato & parsnip mousseline, lollipop kale, delicata squa[illegible]
Blood orange gastric

OR
• Barramundi Seabass
Farro verde, Dungeness crab, lotus root, cream of lemon tar[illegible]
** Add on 7gm of Perlita sturgeon caviar +30 **

OR
• Fresh Porcini Tagliatelle
Wild mushroom medley, vintage port reduction, cream sa[illegible]
Aged Parmesan cheese (vegetarian)
** Add on 7gm of shaved Perigord truffle +30 **

• Chestnut Montblanc

### Vegan menu analysis

The chat playground analyzes a photographed menu and identifies cauliflower soup and winter chicory salad as having vegan versions available on request. It suggests asking about the mains and desserts.

[4:48](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=288s) [The chat playground answers a question about vegan menu options, identifying cauliflower soup and winter chicory salad as available in vegan versions on request.](https://assets.ai.engineer/sites/aiecode2025/sha256/a227ef7728af3dc690b7dd51a453a00dadae94ab73d5bf76d33bbd18606a3145.webp)

Barramundi Seabass
Farro verde, Dungeness crab, lotus root, cream of lemon tarragon
** Add on 7gm of Perlita sturgeon caviar +30 **

OR

Fresh Porcini Tagliatelle
Wild mushroom medley, vintage port reduction, cream sauce, aged parmesan cheese (vegetarian)
** Add on 7gm of shaved Perigord truffle +30 **

Chestnut Montblanc

OR

Apple, Pear & Rhum Raisin Spiced Crumble

Is there anything vegan on the menu today?

Yes, there are vegan options available on the menu. The Cauliflower Soup and Winter Chicory Salad both mention that vegetarian and vegan versions are available upon request. However, the main course and dessert options do not specify vegan versions, so you might want to ask the restaurant if they can accommodate vegan versions of those dishes as well.

Interface text:
All projects / cvi-ai-westus1 / Chat playground
Project cvi-ai-westus1
View Code
Prompt flow
Deploy to a web app
Launch
Import
Prompt samples
Clear chat
Chat capabilities
Show JSON
Add your data PREVIEW
Parameters
'Apply changes' button below to apply your changes.
Reset to default

### French menu interpretation and translation

Asked what’s good on the menu, the chat lists French dishes with English translations and says the choice depends on personal taste.

[7:38](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=458s) [Chat response lists French menu items with English translations under starters, main courses, and cheese or dessert.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f2dabba28d296151b65913050b4cea4c069dc4dc2e89f8ac4f8448b94fb2de5.webp)

The menu board in the image presents a variety of French dishes for a set price of 19 euros, which includes coffee. Here's what's listed:

Entrées (Starters):
• Terrine de lapin aux noisettes (Rabbit terrine with hazelnuts)
• Salade composée (Mixed salad)
• Filets de hareng pommes de terre à l'huile (Herring fillets with potatoes in oil)
• Œufs durs mayonnaise (Hard-boiled eggs with mayonnaise)
• Rillettes du MANS (Rillettes from Le Mans)

Plats (Main Courses):
• Plat du Jour (Dish of the day)
• Poulet grillé frites (Grilled chicken with fries)
• Côte de porc grillée, haricots verts (Grilled pork chop with green beans)
• Bavette à l'échalotes frites (Flank steak with shallots and fries)
• Escalope de volaille à la crème tagliatelles (Poultry escalope with cream and tagliatelle)

Fromage ou dessert (Cheese or Dessert):
• Camembert ou chèvre (Camembert or goat cheese)
• Crêpe sucrée ou confiture (Sweet crepe or with jam)
• Glace 2 boules (Ice cream, two scoops)
• Crème caramel (Caramel custard)
• Fromage blanc (Fresh cheese)

The choice of what's "good" would depend on your personal taste preferences, but this menu offers a variety of traditional French [illegible]

Interface text:
Clear chat
Chat capabilities
Show JSON
[illegible] data PREVIEW
Parameters
[illegible] button below to apply your changes.
[illegible]eset to default

[8:00](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=480s) [Azure AI Studio shows a question about a menu image and a response translating French starters.](https://assets.ai.engineer/sites/aiecode2025/sha256/3608a5f99d53ea01ced083adb1954d64868fc56035abaa10ba536ccac3b9c19d.webp)

Menu image:
Crème caramel
Fromage blanc
Bon Appétit
alamy
www.alamy.com

What's good on the menu today?

The menu board in the image presents a variety of French dishes for a set price of 19 euros, which includes coffee. Here's what's listed:

Entrées (Starters):
• Terrine de lapin aux noisettes (Rabbit terrine with hazelnuts)
• Salade composée (Mixed salad)
• Filets de hareng pommes de terre à l'huile (Herring fillets with potatoes in oil)
• Œufs durs mayonnaise (Hard-boiled eggs with mayonnaise)
• Rillettes du MANS (Rillettes from Le Mans)

Plats (Main Courses):
• Plat du Jour (Dish of the day)
• Poulet grillé frites (Grilled chicken with fries)

Interface text:
Microsoft Edge
File  Edit  View  History  Favorites  Tools  Profiles  Tab  Window  Help
Wed Jun 26 12:38 PM
Work
AI Demos
Chat playground - Azure AI Studio
https://ai.azure.com/playground/chat?wsid=[illegible]
Azure AI Studio / All hubs / cvi_ai-westus1 / All projects / cvi-ai-westus1 / Chat playground
Project: cvi-ai-westus1
To all projects
Current project: cvi-ai-westus1
Project overview
Settings
Get started
Model catalog
Model benchmarks
Prompt catalog
Project playground
Chat
Assistants PREVIEW
Images
Completions
Tools
Code PREVIEW
Prompt flow
Tracing PREVIEW
Evaluation PREVIEW
Fine-tuning PREVIEW
Components
Data
Indexes
Deployments
Content filters PREVIEW
Chat playground
Export  View Code  Launch  Import  Prompt samples
Deployment *
gpt-4-vision
System message  Add your data PREVIEW  Parameters
Please select the 'Apply changes' button below to apply your changes.
Apply changes  Reset to default
System message
Add section
Clear chat  Chat capabilities  Show JSON
Type user query here. (Shift + Enter for new line)
543/128000 tokens to be sent

### Menu translation: mains and desserts

The response translates the French main courses and desserts into English, then says what is “good” depends on personal taste.

[8:06](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=486s) [Close-up of the response listing French main courses and desserts with English translations.](https://assets.ai.engineer/sites/aiecode2025/sha256/961dba59d2c01b0a17d7bf37c963d5a0f7487050a13b20284f8fdea7f737bcb2.webp)

Plats (Main Courses):
• Plat du Jour (Dish of the day)
• Poulet grillé frites (Grilled chicken with fries)
• Côte de porc grillée, haricots verts (Grilled pork chop with green beans)
• Bavette à l'échalotes frites (Flank steak with shallots and fries)
• Escalope de volaille à la crème tagliatelles (Poultry escalope with cream and tagliatelle)

Fromage ou dessert (Cheese or Dessert):
• Camembert ou chèvre (Camembert or goat cheese)
• Crêpe sucrée ou confiture (Sweet crepe or with jam)
• Glace 2 boules (Ice cream, two scoops)
• Crème caramel (Caramel custard)
• Fromage blanc (Fresh cheese)

The choice of what's "good" would depend on your personal taste preferences, but this me[illegible]
to different tastes. The inclusion of both meat and fish options for the main course, as well [illegible]
provides a well-rounded meal experience. The phrase "Bon Appétit" at the bottom wishes [illegible]

Interface text:
to apply your changes.
Type user query here. (Shift + Enter for new line)

### Fallen utility poles image demo

For an energy-industry use case, the presenter attaches a photo of fallen electric poles to GPT-4o and asks an open-ended question about the scene.

[8:49](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=529s) [Azure AI Studio shows an image of fallen utility poles attached to the question, “What's going on here?”](https://assets.ai.engineer/sites/aiecode2025/sha256/1431cc36b90eed83d0b4936349e20dfee04bbbf60288130b472ac8ed8418b421.webp)

Chat playground

Deployment *
gpt-4o

System message
You are an AI assistant that helps people find information.

Start chatting
Test your assistant by sending queries below. Then adjust your assistant setup to improve the assistant's responses.

What's going on here?

Interface text:
Azure AI Studio
All hubs / cvi_ai-westus1 / All projects / cvi-ai-westus1 / Chat playground
To all projects
Current project: cvi-ai-westus1
Project overview
Settings
Get started
Model catalog
Model benchmarks
Prompt catalog
Project playground
Chat
Assistants PREVIEW
Images
Completions
Tools
Code PREVIEW
Prompt flow
Tracing PREVIEW
Evaluation PREVIEW
Fine-tuning PREVIEW
Components
Data
Indexes
Deployments
Content filters PREVIEW
Export
View Code
Prompt flow
Deploy to a web app
Launch
Import
Prompt samples
Add your data PREVIEW
Parameters
Apply changes
Reset to default
Add section
Clear chat
Chat capabilities
Show JSON
70/128000 tokens to be sent
Microsoft Edge
File Edit View History Favorites Tools Profiles Tab Window Help
Wed Jun 26 12:39 PM
Work
AI Demos
Chat playground - Azure AI Studio
https://ai.azure.com/playground/chat?wsid=[redacted]

[9:05](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=545s) [A photo of fallen utility poles is attached to the question, “What's going on here?”](https://assets.ai.engineer/sites/aiecode2025/sha256/106989d4b323d9aadae687d89549c7dd3045b9103ccb2e92fc39bd1c5a44c076.webp)

What's going on here?

### AI describes utility infrastructure damage

GPT-4o describes fallen utility poles and potential hazards in a photo. The presenter suggests using image analysis to monitor infrastructure.

[9:27](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=567s) [A chat response describes fallen or leaning utility poles and possible hazards beneath a partially visible roadside photo.](https://assets.ai.engineer/sites/aiecode2025/sha256/690630d1900be7979e74c1e3c5d43ef4de71d41e310e157135547a3fa47b2da1.webp)

What's going on here?

The image shows several power lines and utility poles that have fallen or are leaning, indicating damage to the infrastructure. This kind of damage can be caused by severe weather events such as storms, high winds, or possibly even a vehicle collision. Utility workers and repair crews may be present to assess the damage and begin repairs to restore power and ensure safety. The scene appears to be a roadside area with significant disruption to the power supply and potential hazards from downed electrical lines.

Interface text:
Type user query here. (Shift + Enter for new line)
246/128000 tokens to be sent

### Follow-up question about electricity

The chat judges electricity unlikely to be working after describing fallen power lines and leaning poles. The speaker suggests JSON output for automated monitoring.

[10:28](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=628s) [The chat answers a follow-up question by saying electricity is highly unlikely to be working in the pictured area.](https://assets.ai.engineer/sites/aiecode2025/sha256/5762623a2e44c89a29e73ed5b134272b3712f1f2f5ad8aa53dc2489efec90b45.webp)

What's going on here?

The image shows several power lines and utility poles that have fallen or are leaning, indicating damage to the infrastructure. This kind of damage can be caused by severe weather events such as storms, high winds, or possibly even a vehicle collision. Utility workers and repair crews may be present to assess the damage and begin repairs to restore power and ensure safety. The scene appears to be a roadside area with significant disruption to the power supply and potential hazards from downed electrical lines.

Is the electricity working here?

It is highly unlikely that the electricity is working in the area shown in the image. The downed and leaning utility poles, along with the fallen power lines, suggest significant damage to the electrical infrastructure. This kind of damage typically results in power outages, as the continuity of the electrical grid is disrupted. Repair crews would need to assess and fix the damage before power can be safely restored.

Interface text:
Type user query here. (Shift + Enter for new line)
324/128000 tokens to be sent

### Azure AI model catalog

The catalog displays 1,674 models with Serverless API selected in the deployment options menu. The presenter contrasts serverless deployment with using managed compute.

[14:17](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=857s) [Model catalog with the deployment options menu open and Serverless API checked.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba4161c88d4e4b2894d8de5703ae9144e8d1389663239dc2e3b6d76063543d57.webp)

Find the right model to build your custom AI solution
Show filters

Announcements

AI21 Jamba-Instruct is here!
AI21's cost-effective model optimized for long context, now available on Azure.
View models
Read blog

Phi-3 Serverless APIs are now available!
Microsoft's latest Phi-3 SLMs offer groundbreaking performance at a small size
View models
Read blog

GPT-4o is now available!
GPT-4o, the most advanced multimodal model from OpenAI, now available on Azure
View models
Read blog

All filters
Collections
Deployment options
Inference tasks
Fine-tuning tasks
Licenses

Deployment options
Managed compute
Serverless API [checked]

Search
Models 1674

gpt-4o — Chat completion
dall-e-3 — Text to image
gpt-35-turbo-instruct — Chat completion
davinci-002 — Completions
text-embedding-ada-002 — Embeddings
gpt-4-32k — Chat completion
gpt-35-turbo-16k — Chat completion
gpt-35-turbo — Chat completion
babbage-002 — Completions
Phi-3-mini-4k-instruct — Chat completion
Phi-3-vision-128k-instruct — Chat completion
Phi-3-small-8k-instruct — Chat completion
Phi-3-small-128k-instruct — Chat completion
Phi-3-mini-128k-instruct — Chat completion
Phi-3-medium-4k-instruct — Chat completion
Phi-3-medium-128k-instruct — Chat completion
Meta-Llama-3-8B-Instruct — Chat completion
Meta-Llama-3-8B — Text generation
Llama-2-7b-chat — Chat completion
Llama-2-7b — Text generation
Meta-Llama-3-70B-Instruct — Chat completion
Llama-2-70b-chat — Chat completion
Llama-2-70b — Text generation
Llama-2-13b-chat — Chat completion
Llama-2-13b — Text generation
CodeLlama-7b-hf — Text generation
CodeLlama-7b-Python-hf — Text generation
CodeLlama-34b-hf — Text generation
CodeLlama-34b-Python-hf — Text generation
CodeLlama-34b-Instruct-hf
CodeLlama-13b-hf
CodeLlama-13b-Instruct-hf
CodeLlama-7b-Instruct-hf
CodeLlama-13b-Python-hf

Interface text:
Microsoft Edge
File Edit View History Favorites Tools Profiles Tab Window Help
Wed Jun 26 12:44 PM
Work
AI Demos
Model catalog - Azure AI Studio
https://ai.azure.com/explore/models?tid=72f988bf-86f1-41af-91ab-2d7cd011db47
Azure AI Studio / Model catalog
Home
Get started
Model catalog
Model benchmarks
Prompt catalog
Azure OpenAI
AI Services
Management
All hubs
Resources and keys
Quota
Invite

## Full transcript

[0:00](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=0s) [on hold music] Uh, so, um, I'm Cédric Vidal.

[0:16](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=16s) I'm an, uh, principal AI advocate at Microsoft. Uh, and today, uh, I'm going to, uh... We're going to do quite cool stuff. Uh, we're going to do-- to talk about multi many things.

[0:29](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=29s) Uh, multi, uh, models, multimodality, multilingual, multi-agents. Uh, all of this with, uh, Azure AI. Um, so yeah. And also one particularity is, uh, apart from a few slides at the beginning, it's only demos.

[0:45](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=45s) Uh, and to be honest, bear with me in case, uh, one of them or all of them don't work. [laughs] Uh, but, uh, it's gonna be fun. We'll see how it goes.

[0:55](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=55s) Um, so as you know, you know, Azure AI is, uh, the best AI platform, uh, out there. Uh, we have, um, a lot of, uh, AI, uh, services. Uh, we can do, uh, machine learning.

[1:09](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=69s) Uh, and we also, uh, do all of that responsibly with the whole responsible AI framework. And we encapsulate all of this, um, in the Azure AI Studio, and I'm going to do, uh, a lot of demos, uh, of i-- Azure AI Studio today.

[1:28](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=88s) And, um, since now almost a year, a bit more than a year, um, we've been, uh, partnering with OpenAI, of course. Uh, and we have all of the OpenAI models available on Azure on the Azure platform.

[1:41](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=101s) But, uh, we're gonna see that, uh, in addition to all the Mo-- OpenAI models that we have and all the modalities that we can get using those, we also have many more models available o-on the platform for, um, uh, text of course, uh, vision, uh, and speech.

[2:01](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=121s) Um, and many organizations trust us today, uh, to, uh, use, uh, AI and build their products. Um,

[2:11](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=131s) so without, um, further ado. So, uh, I'm going to jump in the demos very quickly. But before I do, so we've had many things announced at Build a couple of months ago.

[2:24](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=144s) Uh, we've had, uh, the GA version of Azure AI Studio. We've had, uh, the latest, uh, model from OpenAI, GPT-4, uh, Omni, uh, which supports, uh, text, vision, uh, and soon speech.

[2:37](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=157s) Uh, we've had the new small la-language models, um, from Microsoft, from MS Research called Phi-3. Um, now we have also announced GPT-4 Turbo with Vision, uh, DALL·E 3, and Whisper.

[2:51](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=171s) Um, we've announced the Assistants, uh, API that allows to, uh, build, uh, agents, and I'm going to demo it. Uh, we've announced the fine-tuning for GPT-4, the new, uh, inference batch API.

[3:06](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=186s) Um, and um, also another very cool thing I'm going to demo today, and you had a glimpse of it. Uh, I mean, I guess the surprise is kind of, um, uh, out, uh, but the video, video translation service that I'm going to demo.

[3:21](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=201s) Um, and Azure AI Studio. Oh, yeah. So let's go straight to the demos now. Um, so apart from those slides, now it's only demos.

[3:32](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=212s) So the fun begins. Um, okay. The first demo. So we've had, um, Azure. I mean, uh, uh, like a year ago when, uh, everything started, uh, you didn't have much, uh, many choices.

[3:48](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=228s) It was basically, uh, GPT or, or GPT. The, the only modality available was text. Um, but now things have changed dramatically. Now, we support also, uh, multimodal vision, mixing text and vision, and this opens a completely new era of use cases.

[4:07](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=247s) For example, here and, um, let me zoom. So I'm going to demo, uh, GPT-4, um, o. And actually I selected Vision here, but what I wanted to select was GPT-4o.

[4:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=260s) Um, and I'm going to demonstrate a use case where... So it is kind of small right now, but, uh, this is, um, a menu from a restaurant. Um, and I'm going to ask what's vegan on the menu today.

[4:38](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=278s) So here we can see the menu a bit better.

[4:42](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=282s) So as you can see, we have, uh, winter chicory salad, uh, duck, sea bass, et cetera. Um, and what's very interesting here is that, uh, the menu that you just saw was, um, the, the font is funny, but it's printed.

[4:58](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=298s) Uh, so it's a font from a computer. Uh, and GPT-4o does a very good job at reading what's on the menu. Um, and let me zoom here so that we can see what's...

[5:11](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=311s) So I asked whether there were vegan options today on the menu. And what's interesting is that it looks, it mixes vision, so it extracted all the text from the image.

[5:23](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=323s) But not, not only that, but it reasons on it. So it analyzed all the items on the menu today and for each one of them, uh, looked at which ones were, uh, vegan.

[5:33](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=333s) And as you can see here, uh, the cauliflower soup and winter, um, uh, chicory salad. Uh, let me zoom. Ah.

[5:44](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=344s) Up. Okay. So, okay. Both mention, uh, vege-vegetarian and vegan versions. Um, so that's a very good example of how to mix, uh, text and reasoning.

[6:00](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=360s) Something that was not possible before with just OCRs, which becomes available with the new generation of multimodal, um, models.

[6:10](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=370s) Another example, slightly harder because this one has

[6:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=380s) handwritten text. So this menu has not been printed. It has been written by hand on a chalkboard.

[6:33](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=393s) So as you can see... Oh, and it's in French. Um, so not only does it recognize handwritten, um, uh, sentences written on a chalkboard in a picture, but it also translates it and reasons on it.

[6:54](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=414s) So that's three things that the model is doing all at once, thanks to, uh, multimodality. Uh, this is very important to understand how it differs from what we were doing before, because before we were using image to text to extract the text and then reason on it.

[7:12](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=432s) Now, the model understands natively both pixels and text, and in its internal, um, representation has the same vectors for the same concepts, visual concept and textual concepts.

[7:30](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=450s) That's a very important, um, thing to understand. And as you can see here,

[7:41](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=461s) it displays, um, the answer. Uh, I mean, the, the... I, I ask what's on the menu today. So it's displaying the, um, entries of the menu in French with the English, uh, translation because I asked the question in English.

[7:57](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=477s) Uh, and I could also... Oh, yeah, I asked-- Okay, that's funny. Um, because I asked what's good on the menu today. And so

[8:07](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=487s) the choice of what's good would depend on your personal taste preferences. [laughs]

[8:12](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=492s) Uh, and, uh, the menu offers a variety of traditional French dishes that could cater anyway, um, today. So yeah. So let's move on now to the next, uh, demo.

[8:24](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=504s) So we looked at, uh, you know, something that you might wanna do on the-- at the restaurant, uh, when you are in a foreign country and you don't understand what's in the menu, and you want to u- to get a better understanding, uh, if you have a, a special diet.

[8:37](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=517s) So that's very, uh, convenient. But that technology can also be used for more serious, um, challenges, uh, or use cases. So in this case,

[8:50](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=530s) we're going to look... And that's actually, uh, an actual use case, um, fr- from a discussion I had a couple of weeks ago with a customer working in the energy industry.

[9:01](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=541s) Um, and so here we have, um, a picture of electric poles, uh, that fell on the ground. And I can ask a very open question. What's going on here?

[9:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=560s) By the way, you can see how fast the model replies, which is quite something.

[9:27](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=567s) So not only is GPT-4o, um, understanding both images and text, but it's also much faster, uh, at, uh, answering. So as you can see here, the image shows several power lines and utility poles that have fallen or are leaning, indicating damage to the infrastructure.

[9:45](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=585s) I'm not gonna read everything, but what matters here is if I was, uh, working in the energy transport industry, uh, I might want to observe, um, continuously all the infrastructure of all the networks, like for a whole country, um, uh, at the edge to make sure that the network is operational.

[10:07](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=607s) So I might want to automate looking at all the video cameras, uh, of, uh, filming the, um, the, the infrastructure. So I could ask, "Is the electricity working here?"

[10:28](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=628s) It is highly unlikely that the electricity is working in the area shown in the image. Of course, here I ask the question in natural language, and the answer is presented to me in natural language too.

[10:40](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=640s) But I could also ask for the output to be generated in JSON in a format that could be interpreted by code so that I can automate, um, like dashboards and monitoring of, uh, re- infrastructure in real time.

[10:58](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=658s) Um, another use case is for the insurance industry. So here we have a house that we can ask what happened.

[11:13](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=673s) The image shows a house that has collapsed and is severely ti- tilted. Uh, uh, natural disasters such as hurricanes, earthquakes, or landslide. Um, so yeah, that's also a very interesting use case for the insurance industry.

[11:36](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=696s) Next. So, uh, so next time-- The next one is not gonna be a surprise because there was a spoiler. But, um, so we talked about, um,

[11:46](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=706s) multimodal models, but now we're gonna talk about another modality, uh, speech. Uh, the, um, Azure AI, uh, team, uh, product team has released, uh, an amazing new feature which allows you to translate, uh, videos.

[12:04](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=724s) So here I'm gonna play that video. So disclaimer, that's me on the video. [laughs]

[12:11](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=731s) Um, this is our new video translation service. With this, I can translate videos into other languages in my own voice. Jetzt kann ich Deutsch sprechen, wie ich es immer wollte.

[12:24](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=744s) Me hubiera gustado saber hablar español, pero ahora puedo hablarlo sin haber aprendido el idioma. Posso anche sussurrare in italiano. そして日本語で大きな声で話してください。 This will make the world more inclusive.

[12:39](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=759s) So what's really impressive about, um, that video is not only the fact that now I can speak German. [laughs] But [laughs], um, it took into consideration, uh, the intonation of what I was saying.

[12:54](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=774s) So when I was whispering, it was whispering, too. When I was yelling, it was yelling, too. Um, so it, it takes into account, um, the, the language and, and the tone.

[13:07](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=787s) A disclaimer, I stitched the different videos together myself using post-processing. But apart from that, I didn't do anything. Like, the, the, the service did that all by itself.

[13:19](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=799s) Um, now... Yes? Um, so as I understand, these are, uh, these are, uh, native models. Uh, do these success embeddings models as well? Uh. What is success embeddings? Okay, I'm gonna talk about embeddings models in a minute.

[13:31](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=811s) Okay. [laughs] Um, so, tak, tak, tak, where was I? Uh, model catalog. And thank you. That's a good segue, actually, because... So like I was saying, like, a, a, a year ago when GPT-4, ChatGPT was released, uh, basically you had almost no choice.

[13:51](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=831s) Now, the amount of models available on the Azure AI model catalog is extraordinary. Um, so here, uh, I'm gonna remove, uh, that, uh, filter here so that we display all of them.

[14:07](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=847s) So as you can see here, we have 1,600 models available in the model catalog right now. And what I like, there is one specific feature that I really like, is deployment options.

[14:23](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=863s) Here you can select serverless API. So you have two ways to de- to deploy, uh, models on Azure AI at the moment. You can deploy them serverless or you can, uh, deploy them, uh, using your infrastructure.

[14:39](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=879s) Uh, bring your own infrastructure means basically that you, uh, rent for GPUs, that you pay for GPUs whether you use the endpoint or not. Uh, serverless means that you pay for the token and that the infrastructure is, uh, managed for you by the vendor.

[14:57](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=897s) Um, and paying by the token is nothing new. You've been doing that with OpenAI GPT, uh, ever since it was released. But now you can do it for many vendors on the marketplace.

[15:08](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=908s) And as you can see here, those are all the vendors and all the models that are available on the catalog right now serverless. So you pay by the token and you have n- nothing to, um, manage yourself.

[15:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=920s) Uh, not to mention the fact that, um, getting GPUs right now is not the easiest. Uh, so being able to use those models serverless is... makes it much easier.

[15:31](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=931s) Um, and because we have so many models now, it- it's kind of hard to know which one to use. So now we also have the model benchmarks where we compare, not all, but we compare many of the models that are available in the catalog.

[15:46](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=946s) And you can look at the accuracy as, as, as well as a bunch of other metrics to figure out which model you wanna use for your use case.

[15:58](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=958s) One... Okay, let's hope that the Wi-Fi is not dead. Okay. Uh, one of those models, uh, w- that I want to focus on today, because we talked about GPT-4o, which is a very big model, uh, able to do text and, uh, visual analysis.

[16:18](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=978s) But, and Microsoft Research also came up with our own text and visual multimod- uh, modal model called Phi-3 Vision 128k. Um, this model is very, very interesting.

[16:34](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=994s) It's a family of model. We have a vision version. We have, uh, many sizes. That one specifically is very interesting because, um, I'm going to upload one of the use case.

[16:47](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1007s) For example, randomly, the same use case as the one I, uh, talked about before with GPT-4o. Um, so is electricity, uh, electricity working here?

[17:05](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1025s) And so what's really interesting here, um, is that the model, um, being, despite being much smaller and the explanation is a bit simpler.

[17:22](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1042s) But still, Phi-3 Vision is able to analyze the image and give a very good answer about the fact that the electricity most likely is not working here because of an outage, uh, or disruption.

[17:35](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1055s) So now you have the, uh, possibility in addition to GPT-4o to use Phi-3 Vision for that.

[17:47](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1067s) And not only can you use Phi-3 Vision as a service on Azure AI, we also have in the Phi-3 family a very, very small model, uh, of 3.8, uh, billion parameters, which, uh, weighs under, uh, roughly two gigabytes.

[18:06](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1086s) Uh, and here, if I refresh, um, the, uh, window here, and Dan, I hope I didn't make a mistake. No, it's okay. And so model size two gigabytes. And here, Phi-3 3.8 gigabytes quantized with four bits is run- is downloading in the browser and running locally on the edge using WebGPU,

[18:31](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1111s) which is a new HTML5, uh, specification allowing application and browser to get access to the GPU. And so here I can ask, um,

[18:41](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1121s) "What do you know about the Rivian R2?" And you, you know, you're... This is gonna be a segue to one of the next subjects I'm gonna be talking about after.

[19:01](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1141s) And so as you can see, you get an answer generated in the browser at... I mean, look at how fast it is for a model running locally. Um, so okay, uh, I have a Mac which has an Apple silicon and, uh, some kind of GPU support, but it's by no mean a horse- a horsepower, uh, machine.

[19:24](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1164s) Uh, it's a MacBook Pro M2, I believe. Uh, and as you can see, it's running very, very well in the browser. That means that I can disconnect the Wi-Fi.

[19:34](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1174s) So I'm not gonna do it now, but you could disconnect the Wi-Fi and, and it would still run locally. It's, um, which is also very interesting for, um, um,

[19:46](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1186s) you know, uh, to process, uh, sensitive data.

[19:50](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1190s) Next thing, chat. So here, w- if I ask the question, uh, "What are the, uh, different

[20:02](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1202s) Rivian models?" So I'm asking GPT-4 Turbo, um, which was last updated in April 2023. It knows about the Rivian R1T, the R1S.

[20:21](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1221s) Also, uh, Amazon's, um, Rivian truck, and that's all. Now

[20:29](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1229s) I can select an index to do RAG, Retrieval Augmented Generation, where I can ground the, my model into my own documents. And I'm not gonna show that now, but what I did, uh, befo- to prepare the demo, I just took a bunch of Wikipedia pages of Rivian models that I uploaded to the index,

[20:54](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1254s) um, and now I am querying. So if I ask again the same question, "What are the different

[21:04](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1264s) Rivian models?" Huh, is it bugging? [laughs]

[21:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1280s) That's the problem with live demos. Uh, let me clear. I'm gonna copy that.

[21:28](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1288s) I'm gonna copy, clear, and rerun. Okay. Uh, I'm gonna refresh.

[21:46](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1306s) Okay. I have the index still selected. Now I should be able to... Huh.

[21:53](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1313s) Um, is it the model? Let's try with Phi-3 Turbo.

[22:02](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1322s) Yeah. So now, um, so apparently we have a, a bug with the other model, but, uh, with Phi-3 Turbo it works. And here you can see that we have the R1T, the R1S, the newly released R3, and we should have...

[22:17](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1337s) It doesn't mention the R2, but um, but it me- it does mention the R3. Um, so if I ask, "What about the R2?"

[22:31](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1351s) Yeah. So it does know about it. Um, so this is very interesting because it allows to use an LLM on up-to-date, uh, information. Now, let's move on to the next demo.

[22:45](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1365s) Um, so I'm gonna skip that one, but what I can show here is one thing which is very important is evaluation of models. Because when you build a LLM, uh, application, you want to be able to evaluate how good they are.

[23:01](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1381s) And when you make modifications to the system prompt or you change your models on your application, you wanna make sure that it continues to work as expected. So inside Azure AI Studio, you have a feature called Evaluation, which allows you to run, um, a bunch of metrics.

[23:17](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1397s) Here I show coherence, groundedness, and relevance. Groundedness is something that is very important for RAG applications because you wanna make sure that the answer is grounded into the documents.

[23:28](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1408s) So this system allows you to do that monitor- moderately, uh, easily. And as you can see here, so I used a very simple dataset that I prepared for the demo.

[23:38](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1418s) So I have only one entry in my, um, evaluation dataset. But still, it shows, um, that for that question, which is, "What are the different Rivian models?" The coherence was four.

[23:50](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1430s) It was w- well grounded because it's evaluated between one and five.

[23:56](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1436s) Um, next, something very cool that I absolutely want to show you before we run out of time. So here, um, this is one example of how to build an agent with code interpreter.

[24:09](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1449s) Last Sunday, and that's real data from last Sunday, I went kite surfing, uh, in the [REDACTED:location], in the [REDACTED:location]. And I did a pretty good in a session that I recording using my watch.

[24:21](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1461s) And I exported the GPX file from, uh, my session from my watch. And now I can ask question, which I up- up- uploaded the file to Code Interpreter. And now I can ask question about it.

[24:35](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1475s) So I can say, "Hey, how long was I on the water?"

[24:46](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1486s) And so here, the LLM is going to automatically generate Python code and execute it in a sandbox to analyze the file that I uploaded, which is a GPX XML file.

[24:59](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1499s) So it's not a CSV. Usually, when you see those demos, they use CSVs, right? But here, I'm using an XML file, which is more complicated. Um, and you're gonna see that it should work.

[25:12](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1512s) Yeah, 42 minutes. That's roughly how much time I was on the water. And now I can ask,

[25:20](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1520s) "How many turns, how many tacks did I do?"

[25:27](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1527s) So a tack in, uh, sailing is basically a turn. Um, and here, this is a much harder question because asking how many tacks I did requires not only analyzing the GPS coordinates, but also analyzing the d- the angular differentiation, difference between each point and applying a threshold to decide which of the points on the

[25:52](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1552s) path are actually turns. And as you can see here, it replies with 217. And now we can ask,

[26:01](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1561s) "Can you draw my session on a map?"

[26:12](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1572s) Because I wanna see visually what it looks like, right? It's more fun. Um, so while it generate... What's very interesting here is that I have no expertise in GPX.

[26:23](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1583s) Like, I don't know the file format. I could, uh... Like, if I show you what it looks like, it's pretty, um, uh, you know, technical and, uh, it's very hard to parse.

[26:33](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1593s) So here, without any explanation of what the file format is, Code Interpreter figured that out by itself. And here,

[26:48](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1608s) here's the map that it drew. And I'm gonna skip. I could have asked, "Hey, can you please..." And I did that to prepare the demo. "Can you please add draw red crosses for each one of my turns?"

[27:02](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1622s) And here is the result. Can you imagine how powerful that is? Like, I didn't code a single line.

[27:12](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1632s) Um, and 57. Can I show last thing? [laughs]

[27:22](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1642s) Uh, well, re- real quick. Um, here's, uh, something pretty cool that I wanna show you too. Uh, this is GitHub Workspaces, and here I can go to that repository, and I can...

[27:36](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1656s) So it's a preview, pre-access, and I can ask, "Can you add a Java GUI

[27:48](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1668s) frontend?" So this is a demo. Uh, I don't know if you noticed, but that was Python code. And I ask, "Can you generate a Java GUI frontend?" And it's gonna automatically, using an LLM, figure out what is the state of the r- code repository, figure out the, uh, what it contains, what it does not.

[28:10](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1690s) So it's gonna, um, write specifications, and based on the specification, we can ask to generate a plan. And every step of the way, if it makes a mistake, we can ask it to make corrections.

[28:24](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1704s) This is preview. This is not yet available. Uh, but this is incredibly powerful. Uh, and that's upcoming. [claps hands]

[28:34](https://www.youtube.com/watch?v=YcOrV-TLNyQ&t=1714s) And I'm over. [laughs] [upbeat music]
