# The 2025 AI Engineering Report — Barr Yaron, Amplify Partners

Barr Yaron, Amplify Partners

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/mQ7_Zje7WKE-2025-ai-engineering-report) · [Video](https://www.youtube.com/watch?v=mQ7_Zje7WKE)

## The 2025 AI Engineering Report

A survey of 500 respondents shows how AI engineering spans job titles, use cases, and customization methods—and where evaluation and production practices still lag.

### Who counts as an AI engineer?

[0:41](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=41s)

Does AI engineering describe a job title or the work people actually do? The early findings of the 2025 State of AI Engineering Survey give that question a concrete setting. [Barr Yaron](https://www.amplifypartners.com/team/barr-yaron), an investment partner at Amplify backing technical founders, introduces the results with a promise of less biography and more charts. The survey has 500 respondents, including conference attendees and livestream viewers. Software engineers and AI engineers make up the largest group.

![Horizontal orange bar chart of respondents’ work titles, with software engineer and AI engineer the longest bars; the presenter stands to the left.](https://ai.engineer/api/talk-essay-media/9ba813c204bf7f18e52bea252bb6a305d5ae57e9dae7f0742d3ced3cd96a45b7)

Software engineers and AI engineers lead the survey’s work-title distribution.

Then comes a quick test in the room. Yaron asks who actually holds the title AI engineer; few hands rise. She asks people with other titles to raise their hands, then keep them up if they do the same work as AI engineers. The response makes the distinction visible: **the work extends beyond the title**. Yaron expects the label to gain ground as this broad technical community grows.

Her Google Trends observation places the emergence of the term around the launch of ChatGPT: AI engineering barely registered before late 2022, then attracted sustained interest. A new discipline does not necessarily mean inexperienced practitioners. Among surveyed software engineers with at least ten years of software experience, nearly half have worked with AI for three years or less, and one in ten started in the past year. Even veteran developers are learning a new practice.

### LLMs spread across multiple applications

[3:17](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=197s)

More than half of respondents use LLMs for both internal and external applications. For customer-facing products, OpenAI supplies three of the five most-used models and five of the top ten. These rankings capture a historical snapshot preceding **Claude 4** and **GPT-4.1**.

Code generation and code intelligence lead the applications, alongside writing assistance and content generation. But the more consequential pattern is breadth: among respondents using LLMs, 94% use them for at least two use cases and 82% for at least three. Adoption here means using models across several kinds of work, often both inside the company and in products customers use.

![Purple horizontal bars rank ten LLM use cases. Code intelligence and generation leads; notes state that 94% use LLMs for at least two use cases and 82% for at least three.](https://ai.engineer/api/talk-essay-media/55c233fcf943aa3b6cf3c46cf075fdcda9df3ff28b2354b9337654bf08264f77)

LLM use spans code generation, writing, and multiple other applications.

### RAG and fine-tuning both have a place

[4:06](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=246s)

Beyond few-shot learning, **retrieval-augmented generation**, or [RAG](https://arxiv.org/abs/2005.11401), is the most popular system-customization approach: 70% of respondents report using it. Fine-tuning is also more common than Yaron expected, with researchers and research engineers doing it most often.

An open-ended question asks fine-tuners which techniques they use. Forty percent of those responses mention [LoRA](https://arxiv.org/abs/2106.09685) or [QLoRA](https://arxiv.org/abs/2305.14314), which Yaron reads as a strong preference for parameter-efficient methods. The responses also name [Direct Preference Optimization](https://arxiv.org/abs/2305.18290), or DPO, and reinforcement fine-tuning. Supervised fine-tuning remains the most popular core training approach, and respondents describe hybrid approaches as well. The resulting picture is a mix of customization methods rather than a single standard recipe.

### Prompts change faster than models

[5:26](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=326s)

Just as one model integration settles, another release arrives with better benchmarks and a breaking change. Respondents report frequent changes to both models and prompts:

| What changes | At least monthly | Faster cadence |
| --- | --- | --- |
| Models | More than 50% | 17% weekly |
| Prompts | 70% | One in ten daily |

Prompt revision is especially frequent; Yaron jokes that some engineers have not stopped typing since **GPT-4** arrived.

A post from [Simon Willison](https://simonwillison.net/) can be enough to make yesterday’s trusted prompt feel inadequate. Yet 31% of respondents report no structured prompt-management tool. **Frequent revision and explicit management have not advanced together.** The survey does not ask how engineers feel about that gap; Yaron leaves that question for a jokingly proposed 2026 follow-up.

### The multimodal production gap

[6:25](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=385s)

Text is well ahead of image, video, and audio in workplace usage. Yaron calls this the multimodal production gap. The gap persists when the chart expands beyond deployments working well to include models already in production but with less traction. Broadening the definition of production adoption does not erase text’s lead.

The next reveal separates nonusers into two groups: those planning eventual adoption and those with no current plan. Audio has the highest adoption intent: 37% of respondents not currently using audio plan to adopt it eventually. That percentage describes intentions among audio nonusers, not current usage across the whole sample. Yaron anticipates an audio wave and expects better, more accessible models to encourage further adoption.

![Stacked bars compare text, image, audio, and video model usage across four adoption categories. Outlines highlight nonusers, with a note that 37% of audio nonusers plan to adopt.](https://ai.engineer/api/talk-essay-media/f4de76ffbfde4d56284e9cc580bcec4cc75a84c5672689ca6838b891914b110e)

Audio has the highest intent to adopt among respondents not currently using it.

### Agents face a larger production hurdle

[7:48](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=468s)

Before asking about agents, the survey fixes a definition: an **AI agent** is a system in which an LLM controls the core decision-making or workflow. That avoids collecting adoption figures against hundreds of different interpretations. Under this definition, 80% of respondents say LLMs work well at work, while fewer than 20% say the same about agents. This is respondent-reported workplace success, not benchmark accuracy or a success rate restricted to agent users.

Most respondents are not yet using agents, but most at least plan to; fewer than one in ten say they will never use them. The gap between current success and future interest is substantial.

For agents already in production, permissions are more consequential than the adoption label alone. Most have write access, typically with a human in the loop, and some can act independently. An agent allowed to change a system presents a different operational responsibility from one that only reads it. Yaron identifies tool permissioning as an area she wants to understand better as deployments grow.

### Monitoring quality and storing context

[8:52](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=532s)

Production systems need monitoring, but respondents do not rely on one method. The monitoring question allows multiple selections, and most select multiple approaches. Standard observability is used by 60% of respondents, while more than 50% rely on offline evaluation.

Accuracy and quality evaluation likewise combine several methods, including collecting user data and using benchmarks. Human review remains the most popular. For tracking their own model usage, most respondents rely on internal metrics. These are distinct concerns: observing system behavior, judging output quality, and understanding model usage do not collapse into a single measurement.

Context storage adds another infrastructure choice. Dedicated vector databases are used by 65% of respondents. Yaron interprets that adoption as evidence that specialized databases provide enough value over general-purpose databases with vector extensions for many use cases. She then reports 35% primarily self-hosting and 30% primarily using a third-party provider, introducing those figures as being among the dedicated-vector-database group. The denominator is not explicit enough to convert those hosting figures into respondent counts.

### What respondents expect next

[10:02](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=602s)

The survey then turns from deployed systems to opinions and forecasts:

- **AI disclosure:** Most respondents think agents should be required to disclose that they are AI rather than human.
- **Inference-time compute:** Willingness to pay more wins, but only by a narrow margin.
- **Model architecture:** Respondents generally expect transformer-based models to remain dominant in 2030, prompting Yaron’s callback that attention may be all we will need.
- **Open and closed models:** A majority expect open-source and closed-source models to converge.

On a more speculative question, the mean respondent estimate for the share of the US Gen Z population that will have AI girlfriends or boyfriends is 26%. This is a forecast, not measured prevalence; the spoken question gives no forecast year.

### Evaluation hurts; learning remains a shared practice

[11:16](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=676s)

Asked for the single most painful part of AI engineering, respondents put **evaluation first**. Yaron connects that result to the conference’s emphasis on evals and the preceding talk: the attention is responding to a problem practitioners already feel.

The closing survey question asks which podcasts and newsletters respondents actively learn from at least once a month, allowing multiple selections. The slide presents the top ten in each category. [Latent Space](https://www.latent.space/) heads both lists, and Yaron singles out Swyx for appearing in both. The lists offer a view of this community’s learning habits, rather than a population-wide ranking of publications.

![Full-screen slide with two numbered top-ten lists: newsletters on the left and podcasts on the right. Latent Space appears first in each.](https://ai.engineer/api/talk-essay-media/b662de37a87397a8d7ce0aedfc298d1d008aac83dc24437aff9e8775c8ab14ac)

Respondents’ top newsletters and podcasts, with Latent Space heading both lists.

At the time of the presentation, Yaron says the full report will appear the following week, adding detail on respondents’ favorite models and tools across the stack. She closes by inviting further discussion of AI trends online and in the conference hallways. The early findings leave evaluation as the leading practical pain point, alongside a community still exchanging the knowledge needed to address it.

## Resources

- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) — The original paper explaining fine-tuning with small trainable matrices while keeping base-model weights frozen.
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) — Combines low-rank adapters with a quantized base model to reduce fine-tuning memory requirements.
- [Direct Preference Optimization](https://arxiv.org/abs/2305.18290) — Introduces a direct training objective for aligning language models with preference data.
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) — Foundational research on combining retrieved knowledge with language generation.
- [Simon Willison’s Weblog](https://simonwillison.net/) — Technical writing and experiments about language models, programming, and AI tools.
- [Latent.Space](https://www.latent.space/) — An AI engineering publication featured in the talk’s learning-source discussion.

## Slides

### AI Engineer World's Fair

[0:01](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=1s) [AI Engineer World's Fair title in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4c3e7728a88b8798506b6838bed4d49472f37a0c8e01c015a7d05d7e0845c9b.webp)

AI Engineer
World’s Fair

### Innovation partner and platinum sponsors

[0:11](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=11s) [AWS as innovation partner above six platinum sponsor logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/80d48bd7e28c2dc32376591d2c85247d90968a7d1a450caab88d207fd0849948.webp)

INNOVATION PARTNER
aws

PLATINUM SPONSORS
Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Survey respondents by work title

Among 500 survey respondents, software engineer and AI engineer were the largest work-title groups. Founder was next, followed by a mix of management, research, product and other roles.

[1:53](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=113s) [Bar chart showing software engineer and AI engineer as the largest work-title groups.](https://assets.ai.engineer/sites/aiecode2025/sha256/454a579f493d2eb79901d18603fc7b0b68592cb686b7750915185f460c1ef852.webp)

Survey demographics: which option best describes your work title?

The largest group of respondents identified as engineers — either software or AI.

Software Engineer
AI Engineer
Founder
Management
VP / CEO / CTO
ML or MLOps Engineer
Researcher
Product Manager
Other
Data Scientist

X-axis ticks: 0%, 5%, 10%, 15%, 20%

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World's Fair
Engineering the future of AI

### Interest in AI engineering over time

US Google Trends interest in “AI Engineering” stayed low until late 2022, then climbed to its highest point at the end of the five-year chart.

[2:46](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=166s) [Google Trends chart showing rising US search interest in AI engineering over five years.](https://assets.ai.engineer/sites/aiecode2025/sha256/4aaebeef16be5ee5c3069c23b7e7da4419ab65107cb8d7588174e9d2096f3f76.webp)

AI Engineering
Search term

United States
Past 5 years
All categories
Web Search

Interest over time
Y-axis ticks: 25, 50, 75, 100
Timeline: May 31, 2...; Dec 5, 2021; Jun 11, 2023; Dec 15, 2024
Note

Amplify AI Engineering Survey
© 2025

Interface text:
Google Trends
Home
Explore
Trending Now
Compare

AI Engineer World’s Fair
Microsoft
aws
smol ai

### Software and AI experience among respondents

AI/ML experience skews toward 0–3 years, while software experience peaks at 10+ years. Among respondents with 10+ years in software, 45% have three years or less of AI/ML experience and 10% have one year or less.

[2:49](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=169s) [Bar chart comparing years of software and AI experience, with notes on respondents who have at least ten years of software experience.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c4f395eb51e1203714f393e3a258f5d0a86c516c0719da51af84562e08a7487.webp)

% of respondents by years of experience in software and AI

Y-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%
Years of experience: 0-3, 4-6, 7-9, 10+
Legend: In Software; In AI/ML

Of the folks surveyed with 10+ years of software experience:
45% have 3 years or less of AI/ML experience.
10% have 1 year or less of AI/ML experience (!)

Amplify AI Engineering Survey
© 2025

Event branding: AI Engineer World’s Fair; Microsoft; aws; smol ai

### GenAI deployment in the workplace

More than half of respondents use LLMs for both internal and external use cases. OpenAI supplies 3 of the top 5 models respondents use in customer-facing products.

[3:19](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=199s) [Bar chart of internal and external GenAI deployment. More than half of respondents use LLMs for both; the slide also highlights OpenAI models used in customer-facing products.](https://assets.ai.engineer/sites/aiecode2025/sha256/e33c792e9a7d240cf5ec80f60bde85852e899c7a91da3f8117fcbf56f3e88713.webp)

Where is GenAI deployed to production in your workplace?

Internal Tools
External Tools
Internal + External Tools
Nothing in production yet, but planning to soon

X-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%

More than half of respondents are using LLMs for both internal and external use cases.

3 out of the top 5 models – and half of the top 10 models – that respondents use for customer-facing products are from OpenAI.

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Microsoft
aws
smol ai

### Primary LLM use cases

Code intelligence and generation leads the surveyed use cases, followed by writing assistance. Among participants who use LLMs, 94% use them for at least two use cases and 82% for at least three.

[3:39](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=219s) [Bar chart showing code intelligence and generation and writing assistance as leading LLM use cases. The slide reports that 94% of LLM users have at least two use cases and 82% have at least three.](https://assets.ai.engineer/sites/aiecode2025/sha256/042e03410bb2936c98af7f710d262463c6b3f65fd924a15902124c5d8323dc9d.webp)

What are your primary use cases for LLMs?

Code intelligence and generation
Writing assistant / content generation
Text summarization
Structured data extraction
Workflow / app automation
Search / recommendation systems
Customer support
Metadata generation
Sentiment analysis
Fraud / threat / anomaly detection

X-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%

The most popular use cases for LLMs today are code intelligence and generation and writing assistance.

There’s also major use case heterogeneity:
- 94% of participants who use LLMs use it for at least 2 use cases.
- 82% use it for at least 3.

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

### AI system customization techniques

Few-shot learning leads, with over 70% of respondents using RAG. Fine-tuning and zero-shot learning each reach roughly 40%; Yaron says the amount of fine-tuning surprised her.

[4:17](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=257s) [Bar chart comparing customization techniques; few-shot learning and RAG lead, with over 70% using RAG.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e020e431d36eb360abd2af4ca30a50565fdf27bc4a9057a949d29ebba7223e1.webp)

What techniques are you using to customize your AI systems?

70%+ of respondents are using RAG.

Few-shot learning
RAG
Fine-tuning
Zero-Shot Learning
Pretraining
Other

X-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World's Fair
Microsoft
aws
smol ai

### AI customization and fine-tuning

Few-shot learning and RAG lead the chart; 70%+ of respondents use RAG. Among fine-tuners, 40% mentioned LoRA or QLoRA, and supervised fine-tuning was the most popular core training approach.

[4:54](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=294s) [Customization chart with a callout stating that 40% of fine-tuners mentioned LoRA or QLoRA and that supervised fine-tuning was the most popular core training approach.](https://assets.ai.engineer/sites/aiecode2025/sha256/53b9abd3d24dd4557787c74b048b0ada3bbd1226b470a9eb9a6942c76f030a1a.webp)

What techniques are you using to customize your AI systems?

70%+ of respondents are using RAG.

Few-shot learning
RAG
Fine-tuning
Zero-Shot Learning
Pretraining
Other

X-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%

40% mentioned LoRA/QLoRA, indicating widespread preference for parameter-efficient methods.

Most popular core training approach is supervised fine-tuning.

Amplify AI Engineering Survey
© 2025

### How often do you update your models?

Monthly is the most common model update cadence, followed by quarterly. More than 50% update at least monthly; 17% update weekly.

[5:15](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=315s) [Bar chart of model update frequency, noting that over 50% update at least monthly and 17% update weekly.](https://assets.ai.engineer/sites/aiecode2025/sha256/f77b7898e80609de286282311c2c3e40ebaedd8795c9607aa9c665dc248b7421.webp)

How often do you update your models?

- More than 50% are updating their models at least monthly.
- 17% weekly.

Y-axis ticks: 0%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%
Categories: Daily, Weekly, Monthly, Quarterly, Almost never

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

[5:21](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=321s) [Bar chart of model update frequency; more than half update at least monthly and 17% weekly.](https://assets.ai.engineer/sites/aiecode2025/sha256/cba71d6172db8a1fcb98a2d49c4f1e9562cb4f075098d2547feac8067ddd043f.webp)

How often do you update your models?

- More than 50% are updating their models at least monthly.
- 17% weekly.

Y-axis ticks: 0%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%
Categories: Daily, Weekly, Monthly, Quarterly, Almost never

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Microsoft
aws
smol ai

### How often do you update your prompts?

Weekly is the most common prompt-update cadence. 70% of respondents update prompts at least monthly, and 10% do so daily.

[5:38](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=338s) [Bar chart of prompt update frequency; 70% update at least monthly and 10% daily.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b59659289672e10ec14e528fb1afc6925ff1baaf90ea6117b2a26780ccf4277.webp)

How often do you update your prompts?

Teams are updating their prompts much more frequently.
- 70% of respondents are updating prompts at least monthly.
- 10% are doing it daily (!)

Y-axis ticks: 0%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%
Categories: Daily, Weekly, Monthly, Quarterly, Almost never

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

### Current working prompt versus a newly read prompt

A Drake meme contrasts a working prompt with one read about from Simon Willison. Yaron jokes that a single blog post can make a trusty prompt feel inadequate.

[5:59](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=359s) [A meme contrasts a current working prompt with a prompt read about from Simon Willison, over the prompt-update survey slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a81095a1a9ae70d50ec46fbc4a472b6702fd98c551deecb176f4f9259e08cd4.webp)

How often do you update [illegible]

Current working prompt
Prompt I read about from Simon Willison

Teams are updating their prompts much more frequently.
- 70% of respondents are updating prompts at least monthly.
- 10% are doing it daily (!)

Y-axis ticks: 0%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%
X-axis labels: Daily, Weekly, [illegible], Almost never

imgflip.com
Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Microsoft
aws
smol ai

### What do you use to manage your prompts?

Internal tools are the most common approach shown, but 31% of respondents say they have no way to manage their prompts. External tools and spreadsheets are less common.

[6:05](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=365s) [Horizontal bar chart of prompt-management approaches; 31% of respondents do not manage their prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/f700830d8d9aa8357a55d536ee2486b08f97dc9025bfba72dfae78a75478ffc1.webp)

What do you use to manage your prompts?

A full 31% of respondents don’t have any way of managing their prompts.

We built an internal tool
We don’t do anything
We use an external tool
We use a simple spreadsheet solution

X-axis ticks: 0%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

[6:08](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=368s) [Bar chart comparing prompt management methods; 31% report doing nothing to manage prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/5189d938ad08d4769c44768949c9bc52fec2da2b5303a6a8514f1eae2dcb5100.webp)

What do you use to manage your prompts?

A full 31% of respondents don’t have any way of managing their prompts.

We built an internal tool
We don’t do anything
We use an external tool
We use a simple spreadsheet solution

X-axis ticks: 0%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

### Model use working well by modality

Among survey respondents, text generation leads by a wide margin for models used at work and working well. Image and audio trail far behind, with video lowest.

[6:31](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=391s) [Bar chart shows text generation working well for far more respondents than image, audio, or video generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/f75eeead987b631f52c701ac1b2355c05fd1676a05b21b01738cf92cff327081.webp)

% of respondents who say they’re using _____ models at work?

Image, video, and audio usage all lag text usage by significant margins.

Y-axis ticks: 0%, 25%, 50%, 75%, 100%
Categories: Text generation / LLMs; Image generation; Audio / voice generation; Video Generation
Legend: Yes, and working well!

Amplify AI Engineering Survey
© 2025

Interface text: AI Engineer World’s Fair; Microsoft; aws; smol ai

### Multimodal production gap

Nearly 80% of respondents say text generation models are working well at work, versus less than 20% for each of image, audio, and video generation.

[6:40](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=400s) [The words multimodal production gap overlay the modality usage chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/0837faa3256de3e326dcc9846a6990d4de60354a8beb662824fe2793a31ad677.webp)

multimodal production gap

% of respondents who say they’re using _____ models at work?

Image, video, and audio usage all lag text usage by significant margins.

Y-axis ticks: 0%, 25%, 50%, 75%, 100%
Categories: Text generation / LLMs; Image generation; Audio / voice generation; Video Generation
Legend: Yes, and working well!

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

### Model adoption intent by modality

Among respondents not yet using audio, 37% plan to adopt it—the highest intent-to-adopt rate across modalities. The chart separates current use by traction and nonuse by adoption plans.

[6:59](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=419s) [Stacked bars show current use and adoption plans across modalities, with a callout that audio has the highest intent to adopt.](https://assets.ai.engineer/sites/aiecode2025/sha256/40be929bafc738e8ec334f9ae5304c47240530ca8428a88029f08ddc7d3debec.webp)

% of respondents who say they’re using _____ models at work?

Audio has the highest intent-to-adopt rate.

Y-axis ticks: 0%, 25%, 50%, 75%, 100%
Categories: Text generation / LLMs; Image generation; Audio / voice generation; Video Generation
Legend:
No, and I don’t currently plan to
No, but I plan to
Yes, but not much traction!
Yes, and working well!

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Microsoft
aws
smol ai

[7:22](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=442s) [Stacked bars compare model use and planned adoption. A callout says 37% of people not using audio plan to adopt it.](https://assets.ai.engineer/sites/aiecode2025/sha256/54e8062af63ab9801a767cb93a86193687554b6e817b3c7d1509b6b080c0de6a.webp)

% of respondents who say they’re using _____ models at work?

Audio has the highest intent-to-adopt rate.

37% of the folks NOT using audio today have a plan to

Y-axis ticks: 0%, 25%, 50%, 75%, 100%
Categories: Text generation / LLMs; Image generation; Audio / voice generation; Video Generation
Legend:
No, and I don’t currently plan to
No, but I plan to
Yes, but not much traction!
Yes, and working well!

Amplify AI Engineering Survey
© 2025

Interface text:
AI Engineer World’s Fair
Engineering the future of AI

## Full transcript

[0:00](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=0s) [upbeat music] [audience applauding]

[0:28](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=28s) All right. Hi, everyone. Uh, thank you for having me here, and huge thanks to Ben, to Swyx, to all the organizers who've put so much time and heart into bringing this community together. [audience applauding]

[0:41](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=41s) Yeah. All right. So we're here because we care about AI engineering and where this field is headed. So to better understand the current landscape, we launched the twenty-twenty five State of AI Engineering Survey, and I'm excited to share some early findings with you today.

[1:03](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=63s) All right. Before we dive into the results, the least interesting slide. Uh, I don't know everyone in this audience, but I'm Barr. I'm an investment partner at Amplify, where I'm lucky to invest in technical founders, including companies built by and for AI engineers.

[1:19](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=79s) And, uh, with that, let's get into what you actually care about, which is enough Barr and more bar charts, and there are a lot of bar charts coming up.

[1:30](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=90s) Okay, so first, our sample. We had five hundred respondents fill out the survey, including many of you here in the audience today and on the live stream. Thank you for doing that.

[1:42](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=102s) And the largest group called themselves engineers, whether software engineers or AI engineers. While this is the AI engineering conference, it's clear from the speakers, from the hallway chats, there's a wide mix of titles and roles.

[1:57](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=117s) You even let a VC sneak in. Um, so let's test this with a quick show of hands. Raise your hand if your title is actually AI engineer at the AI engineering conference.

[2:08](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=128s) Okay, that is extremely sparse. [laughs] Uh, raise your hand-- Put your hands down. Raise your hand if your title is something else entirely. So that should be almost everyone. Keep it up if you think you're doing the exact same work as many of the AI engineers.

[2:27](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=147s) All right, so this sort of tracks. Titles are weird right now, but the community is broad. It's technical. It's growing. We expect that AI engineer label to gain even more ground.

[2:37](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=157s) Uh, couldn't help myself. Quick Google Trends search. Term AI engineering barely registered before late twenty-twenty two. Uh, we know what happened. ChatGPT launched, and the momentum for AI engineering interest has not slowed since.

[2:51](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=171s) Okay, so people had a wide variety of titles, but also a wide variety of experience. Uh, the interesting part here is that many of our most seasoned developers are AI newcomers.

[3:01](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=181s) So among software engineers with ten plus years of software experience, nearly half have been working with AI for three years or less, and one in ten started just this past year.

[3:12](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=192s) So change right now is the only constant, even for the veterans.

[3:17](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=197s) All right, so what are folks actually building? Let's get into the juice. So more than half of the respondents are using LLMs for both internal and external use cases.

[3:27](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=207s) Uh, what was striking to me was that three out of the top five models and half of the top ten models that respondents are using for those external cases for the customer-facing products are from OpenAI.

[3:41](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=221s) The top use cases that we saw are code generation and code intelligence and writing assistant content generation. Maybe that's not particularly surprising. Uh, but the real story here is heterogeneity.

[3:51](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=231s) So ninety-four percent of people who use LLMs are using it for at least two use cases, eighty-two percent using it for at least three. Basically, folks who are using LLMs are using it internally, externally, and across multiple use cases.

[4:06](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=246s) All right. So you may ask, how are folks actually interfacing with the models, and how are they customizing their systems to-- for these use cases? Uh, besides few-shot learning, RAG is the most popular way folks are customizing their systems, so seventy percent of respondents said they're using it.

[4:25](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=265s) The real surprise for me here, I, uh, I'm, I'm looking to gauge surprise in the audience, was how much fine-tune is hap-- fine-tuning is happening across the board. It was much more than I had expected overall.

[4:37](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=277s) Uh, in the sample, we have researchers and we have research engineers who are the ones doing fine-tuning by far the most. We also asked an open-ended question for those who were fine-tuning, what specific techniques are you using?

[4:50](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=290s) So here's what the fine-tuners had to say. Uh, forty percent mentioned LoRA or QLoRA, reflecting a strong preference for parameter-efficient methods, and we also saw a bunch of different fine-tuning methods, uh, including DPO, reinforcement fine-tuning, and the most popular core training approach was good old supervised fine-tuning.

[5:12](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=312s) Many hybrid approaches were listed as well. Um, moving on top, uh, to up-- on top of updating systems, sometimes it can feel like new models come out every single week.

[5:26](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=326s) Just as you finished integrating one, another one drops with better benchmarks and a breaking change. So it turns out more than fifty percent are updating their models at least monthly, seventeen percent weekly.

[5:40](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=340s) And folks are updating their prompts much more frequently. So seventy percent of respondents are updating prompts at least monthly, and one in ten are doing it daily. So it sounds like some of you have not stopped typing since GPT-4 dropped.

[5:54](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=354s) Um, but I also understand. I have empathy. Uh, seeing one blog post from Simon Willison, and suddenly your trusty prompt just isn't good enough anymore.

[6:07](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=367s) Despite all of these prompt changes, a full thirty-one percent of respondents don't have any way of managing their prompts. [laughs] Uh, what I did not ask is how AI engineers feel about not doing anything to manage their prompts.

[6:21](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=381s) So we have the twenty twenty-six survey for that.

[6:25](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=385s) We also asked folks across the different modalities who is actually using these models at work, and is it actually going well? And we see that image, video, and audio usage all lag text usage by significant margins.

[6:41](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=401s) I like to call this the multimodal production gap

[6:45](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=405s) 'cause I wanted an animation. [laughs] Um, and this gap still pers- persists when we add in folks who have these models in production but have not garnered as much traction.

[7:00](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=420s) Okay, what's interesting here is when we add the folks who are not using models at all in this chart, too. So here we can see folks who are not using text, not using image, not using audio, or not using video.

[7:15](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=435s) And we have two categories. It's broken down by folks who plan to eventually use these modalities and folks who do not currently plan to.

[7:24](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=444s) You can roughly see this ratio of no plan to adopt versus plan to adopt. Audio has the highest intent to adopt, so thirty-seven percent of the folks not using audio today have a plan to eventually adopt audio.

[7:39](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=459s) So get ready to see an audio wave. Um, of course, as models get better and more accessible, I imagine some of these adoption numbers will go up even further.

[7:48](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=468s) All right, so we have to talk about agents. One question I almost put in the survey was, how do you define an AI agent? But I thought I would still be reading through different responses.

[7:59](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=479s) Uh, so for the sake of clarity, we defined an AI agent as a system where an LLM controls the core decision-making or workflow.

[8:08](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=488s) Eighty percent of respondents say LLMs are working well at work, but less than twenty percent say the same about agents.

[8:16](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=496s) Agents aren't everywhere yet, but they're coming. Uh, the majority of folks, uh, may not be using agents, but most at least plan to. So fewer than one in ten say that they will never use agents.

[8:28](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=508s) All to say that people want their agents, and I'm probably, uh, preaching to the choir.

[8:34](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=514s) Um, the majority of agents already in production do have write access, uh, typically with a human in the loop, and some can even take actions independently. So, um, excited as more agents are adopted to learn more about the tool permissioning that folks, uh, have access to.

[8:52](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=532s) If we want AI in production, of course, we need strong monitoring and observability. So we asked, "Do you manage and monitor your AI systems?" This was a multi-select question, so most folks are using multiple methods to monitor their systems.

[9:06](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=546s) Sixty percent are using standard observability. Over fifty percent rely on offline eval. And we asked the same thing for how you evaluate your model and system accuracy and quality.

[9:18](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=558s) So folks are using a combination of methods, including data collection from users, benchmarks, et cetera, but the most popular at the, at the end of the day is still human review.

[9:28](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=568s) Um, and for monitoring their own model usage, most respondents rely on internal metrics.

[9:35](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=575s) So storage is important, too. Where does the context live? How do we get it when we need it? Sixty-five percent of respondents are using a dedicated vector database, and this suggests that for many use cases, specialized vector databases are providing enough value over general purpose databases with vector extensions.

[9:53](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=593s) Uh, among that group, thirty-five set-- percent said that they primarily self-host. Thirty percent primarily use a third-party provider.

[10:02](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=602s) All right. I think we've been having fun this whole time, but we're entering a section I like to formally call Other Fun Stuff. Uh, I spent hours workshopping the name.

[10:12](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=612s) So we asked AI engineers, "Should agents be required to disclose when they're AI and not human?" Most folks think, yes, agents should disclose that they're AI. Uh, we asked folks if they'd pay more for inference time compute, and the answer was yes, but not by a wide margin.

[10:28](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=628s) And we asked folks if transformer-based models will be dominant in twenty-thirty, and it seems like people do believe that attention is all we'll need in twenty-thirty.

[10:38](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=638s) Uh, the majority of respondents also think open source and closed source models are gonna converge, so I will let you debate that after. Um, no commentary needed here. So, uh, the average or the mean guess for the percentage of US Gen Z population that will have AI girlfriends, boyfriends is twenty-six percent.

[10:57](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=657s) Um, I don't really know what to say or expect here, but we'll see. Uh, we'll see what happens, uh, [laughs] in a world where folks don't know if they're being left on read or just facing latency issues, um, or, uh, of course, the dreaded, "It's not you, it's my algorithm."

[11:16](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=676s) And finally, we asked folks, "What is the number one most painful thing about AI engineering today?" And evaluation topped that list. Uh, so it's a good thing this conference and the talk before me has been so focused on evals 'cause clearly they're causing some serious pain.

[11:31](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=691s) Okay, and now to bring us home, I'm gonna show you what's popular. So we asked folks to pick all the podcasts and newsletters that they actively learn something from at least once a month, and these were the top ten of each.

[11:43](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=703s) So if you're looking for new content to follow and to learn from, this is your guide. Uh, many of the creators are in this room, so keep up the great work.

[11:52](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=712s) And I'll just shout out that Swyx is listed both on popular newsletter and popular podcast for Latent Space. Uh, so I will just leave this here. [laughs] [audience laughing]

[12:05](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=725s) Um, I think that's enough bar charts and Barr time, but if you wanna geek out about AI trends, you can come find me online, in the hallways. Uh, we're gonna be publishing a full report next week.

[12:14](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=734s) Uh, I'll let Elon and Musk have Twitter today. But, um, it's gonna include more juicy details, including everyone's favorite models and tools across the stack. Thank you for the time.

[12:25](https://www.youtube.com/watch?v=mQ7_Zje7WKE&t=745s) Enjoy the afternoon. [upbeat music]
