# AI Platform Engineering

Patrick Debois

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/5qNXdLbEdew-ai-platform-engineering) · [Video](https://www.youtube.com/watch?v=5qNXdLbEdew)

## AI Platform Engineering: From Pilot Teams to Production

Scaling GenAI across an organization requires shared infrastructure, practical enablement, and governance that helps application teams build and operate useful products.

Familiarity with application development, APIs, and basic deployment practices will help; no model-training background is required.

### What happens after the first AI pilot?

[0:19](https://www.youtube.com/watch?v=5qNXdLbEdew&t=19s)

You have a team experimenting with GenAI. What will it take for the rest of the organization to use it? Patrick Debois approaches that question through an earlier transformation: he organized the first DevOpsDays in 2009, with about 60 attendees, and subsequently watched DevOps spread through enterprises. That experience supplies a useful lens, with a qualification he makes explicit: applying an old paradigm to a new technology can reveal a pattern—or get it completely wrong.

His interest in GenAI starts with **automation intelligence**. Much of his DevOps work involved automating pipelines, making them robust, and ensuring that they delivered value. AI extends that work, but it also raises the familiar question of how a promising capability becomes something an organization can reliably use.

The adoption pattern begins with one pilot team. Two or three more teams try the technology; their experience exposes repeatable patterns; those patterns then support wider adoption. Cloud infrastructure followed this path. Shared abstractions eventually let application teams move faster within their own domains without understanding every part of the underlying infrastructure.

![Orange slide with three groups of blue and yellow circles labeled Incubate, Repeat, and Scale above a Maturity line.](https://ai.engineer/api/talk-essay-media/abb20bd00f794086ec344a1000475ff6bbc57cbde42f2e6991f860ee51db1ca9)

Introducing organizational and technical change: Incubate, Repeat, Scale.

Even the unsettled name of the practice has a function. Debois recalls the arguments over DevOps, alongside earlier movements around agile and ITIL. A label such as **AI engineer** gives people a way to find emerging stories, job postings, and one another. Defining it too rigidly, too early, can restrict what the practice becomes.

### Production work changes the team

[3:59](https://www.youtube.com/watch?v=5qNXdLbEdew&t=239s)

In Debois's experience building GenAI applications over two years, the first organizational friction appeared when the work landed with data scientists who were not accustomed to running production services. Engineers began moving into the data science team. As integration became a larger part of the work, engineering's relative share increased. Capabilities then spread to other teams and eventually moved into the platform.

That movement also reconnected data that had been isolated in a data lake with the applications that needed it. Debois calls this a shift right, contrasting it with security's shift left: responsibilities move as teams discover where the friction actually lies.

Bringing development and operations into the same room was a start, but permanent, intensive collaboration did not scale across an organization. Kubernetes pipelines, infrastructure, and security tooling became shared capabilities. [Team Topologies](https://teamtopologies.com/key-concepts-content/team-interaction-modeling-with-team-topologies), which grew out of [DevOps Topologies](https://web.devopstopologies.com/), provides a vocabulary for this arrangement: feature teams build and run their applications on top of a platform.

The platform team can interact with those teams in three ways:

| Interaction | What the platform team does |
|---|---|
| Service consumption | Provides an API or service the team can use |
| Collaboration | Works with the team to shape the capability it needs |
| Facilitation | Helps the team overcome problems and learn |

**The platform is a product, and application teams are its customers.** Its purpose is to help traditional application developers, supported by data science expertise, deliver AI features within their own teams.

### Model access and reusable data infrastructure

[7:41](https://www.youtube.com/watch?v=5qNXdLbEdew&t=461s)

The first platform capability is **model access**. Instead of every team independently investigating providers and obtaining access, the platform team determines what is appropriate for the company. Existing cloud relationships often influence that choice, making CloudOps a natural collaborator. Debois presents the vendors in this part of the talk as examples, without claiming affiliation.

Retrieval-augmented generation adds a second layer: vector storage and search. Dedicated vector databases appeared early, while traditional database vendors increasingly incorporated vector capabilities. That makes the infrastructure less exotic, but teams still need to understand vectors and embeddings to use it well. The slide places products such as Chroma, Vespa, and Milvus alongside PostgreSQL, Redis, and SingleStore.

![Pink slide titled Unstructured data - Vector Databases, with database logos arranged in two columns, including Chroma, Vespa, Milvus, PostgreSQL, Redis, and SingleStore.](https://ai.engineer/api/talk-essay-media/7048944698b9449b23d7af1f76eb88ad3f2d2fa2e5bc17b224c72721e66c0da0)

Dedicated vector databases alongside databases that support vector search.

A vector database alone does not connect an application to company knowledge. The platform also needs **data connectors** that expose internal sources to multiple teams. Building these once avoids having each feature team rediscover how to integrate the same systems. This ongoing responsibility for connecting and operating retrieval infrastructure is sometimes called RAGOps.

Model version control completes this initial foundation. A central model repository or registry makes versions visible and reusable, much as a shared code library makes existing work discoverable. Teams should not have to invent their own model-storage and discovery practices independently.

### Control access and observe quality

[9:44](https://www.youtube.com/watch?v=5qNXdLbEdew&t=584s)

Larger enterprises often want independence from a particular model provider. A shared proxy can mediate model access while enforcing who may use which services. Debois tentatively compares the OpenAI API's possible role to S3's role in storage: a common interface around which providers might converge. This is a historical forecast in the talk, not a claim of a formal standard or verified compatibility.

Once applications run in production, ordinary request monitoring is insufficient. The platform needs to capture prompts and trace their relationships. Debois describes an initial prompt leading to five or six iterations or questions, so the useful unit of observation is often a sequence rather than a single API call. Providing that tracing centrally saves each team from assembling it independently.

**An available API can still produce poor results.** Production monitoring therefore needs checks on data and output quality, extending practices familiar from MLOps. Continuously running evaluations act as another kind of health check: they can help reveal model changes or unusual user behavior that an uptime check would miss.

Caching and feedback collection are further shared services. Feedback should extend beyond thumbs up, thumbs down, and retry: inline edits reveal how a user would correct the result. Building and managing these capabilities well is expensive enough that duplicating them across teams is wasteful. The growing ecosystem resembles Kubernetes in its breadth; an AI platform is already more than cloud access and a few API keys.

### Help teams experiment—and find a use case

[12:15](https://www.youtube.com/watch?v=5qNXdLbEdew&t=735s)

Providing infrastructure does not automatically make it usable. Like any product team, the platform team must guide its customers and learn from their experience. Enablement is both instruction and a feedback channel: it reveals which capabilities help and which get in the way.

Several forms of support make experimentation easier:

- **Prototyping tools:** Include product owners, who need to explore potential use cases as much as engineers do. Connect those tools securely to company data.
- **Frameworks:** Give developers a way to discover what is possible, supported by education and documentation suited to their existing stacks.
- **Local development:** Let developers experiment in a familiar environment, including while traveling. Improving laptop models can shorten iteration loops, although local stacks have tradeoffs.

These are ways to help teams learn, not merely additional items in a service catalog.

The next obstacle is choosing work worth doing. Mandating GenAI in every part of a product can put implementation ahead of a real use case; Debois compares some of this activity to spending a marketing budget. He also warns that an excessive focus on model training and fine-tuning can postpone getting anything into production.

Premature cost optimization creates a similar distraction. Insisting on a local or cheaper model before establishing the business case can consume engineering effort without producing value. Debois's ordering is to establish the return first, then reduce the cost of delivering it.

Once users provide feedback, someone must interpret it. Engineers accustomed to error stacks may not know what a qualitative correction means for the product. The product owner often holds the domain knowledge needed to distinguish an acceptable answer from a misleading one. Collecting feedback without involving that expertise leaves the improvement loop unfinished.

### Model changes make evaluation essential

[15:34](https://www.youtube.com/watch?v=5qNXdLbEdew&t=934s)

Developer experience includes some unusually difficult choices: which model to start with, how to obtain representative test data, and how much to depend on a fast-changing framework. Debois describes developers first learning through frameworks, then abandoning them after disruptive changes and rebuilding against lower-level APIs. A prompt call may be simple, but tracing, monitoring, and the surrounding integrations still have to work together. His hope is for a dependable tool ecosystem rather than every team rebuilding that machinery.

Debois recalls using approximately eight LLM models over two years while pursuing better quality. Each change required rewriting and reworking application prompts, and the work was manual. From an engineering-management perspective, that combined substantial cost with little assurance that the replacement would behave correctly. **Establish evaluations before refactoring prompts for a new model.**

Evaluation becomes easier to discuss when it starts with checks engineers already understand. A character-count requirement or regular expression has a definite result. For example, a TypeScript check for the talk's twenty-character constraint can be expressed directly:

```typescript
function hasTwentyCharacters(output: string): boolean {
  return Array.from(output).length === 20;
}

const outputs = ["Your order is ready.", "Ready."];
const results = outputs.map(output => ({
  output,
  passes: hasTwentyCharacters(output),
}));

console.log(results);
```

This checks a format requirement; it does not establish whether the answer is useful or correct.

The evaluation ladder then moves beyond deterministic rules:

| Method | What it checks |
|---|---|
| Sentiment helper model | The answer's sentiment |
| Semantic comparison | Relatedness, such as question–answer relevance |
| LLM judge | Qualities assessed by another language model |
| Human feedback | Human assessment of the result |

Using an LLM to judge another LLM introduces a reliability question rather than eliminating one. Debois treats that concern as unresolved, while recognizing model-based judging as a practical tool. The engineering approach combines these methods instead of relying on an informal impression alone.

### Faster generation moves work into review

[18:30](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1110s)

Enablement also has an emotional component. Some engineers initially fear AI; giving them a coding copilot lets them experience its usefulness directly. That puts the team building GenAI capabilities in a second role: helping colleagues become comfortable using AI in their own work.

Debois reports that copilots produced more code, larger AI-generated pull requests, and longer review times. He does not claim that review overhead entirely canceled the generation gains. The point is that speeding up one stage can create friction at the next, so code generation alone is an incomplete account of productivity.

He connects this tension to [Ironies of Automation](https://tc.ifac-control.org/4/1/newsletter/ironies-of-automation/view) and a newer generative-AI treatment of the same problem, [Ironies of Generative AI](https://arxiv.org/abs/2402.11364). The person using automation moves from producing work toward managing and reviewing it. Generation becomes faster while evaluation occupies more of the person's time.

That role change becomes dangerous when the reviewer no longer understands the work well enough to assess it. Without the domain model or hands-on experience, accepting an output can become easier than evaluating it. Debois illustrates this with a weekend Copilot acceptance-rate anecdote; his humorous explanation that developers simply care less is not an established cause. The deeper concern is declining **situational awareness**: less practice producing the work can leave people less prepared to recognize and handle failures.

### Automation creates a duty to prepare for failure

[20:56](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1256s)

The early promise of DevOps could sound like replacing a person's work with a shell script. What grew around that automation was a much larger discipline of preparing for failure. Debois illustrates the balance as 20% automation and 80% failure preparation, an organizing metaphor rather than a measured industry allocation.

The progression matters:

1. **CI/CD and tests** add checks around automated delivery.
2. **Production monitoring** reveals whether the running system is healthy.
3. **Designing for failure** begins by assuming things will break.
4. **Observability** gives operators the situational awareness to understand what is happening.
5. **Chaos engineering** injects failures so teams can keep practicing their response.

Automation removes some work, but these practices help people remain capable when it fails.

The purpose is also to reduce cognitive load during an incident, when people have the least capacity to reconstruct an unfamiliar system. Not every organization adopts every practice; the appropriate investment depends on its risk appetite. But the work of enablement extends to helping engineers feel prepared, not simply making them faster.

![Teal slide maps infrastructure as code, CI/CD, monitoring, resilient engineering, observability, and chaos engineering to automation and failure preparedness, with Reduce Cognitive Load below.](https://ai.engineer/api/talk-essay-media/6db754198bf7952d35cba305a52c0dfc629220587b7a66d75a983107c5be67c7)

20% automation, 80% prepare for failure.

### Governance must survive everyday use

[22:12](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1332s)

Governance starts with personal awareness. A tool that records the screen may promise productivity while also exposing everything visible on it. Teams need to understand what they are copying or transmitting, and how a service handles those inputs. Training opt-outs deserve particular attention because where and when to configure them may not be obvious.

A restricted set of approved models makes license review more manageable. The word open does not by itself establish permission to use a model; its license and origin still need examination. Debois also includes basic awareness of European workflow risk assessment: teams should ask whether a proposed use is allowed and appropriate without expecting every engineer to become a lawyer.

Operational controls require equal care. Debois treats prompt injection as an unsolved problem for which defenses are still necessary. He compares guardrails to intrusion detection and web application firewalls: a growing pile of rules and logs can eventually train people to ignore alerts. Installing a control can create reassurance without ensuring that anyone can operate it effectively.

More uncontrolled text crossing service boundaries also increases the need to scrutinize **personally identifiable information**, with useful metrics and alerting around what is transmitted. These responsibilities bring the platform's remit into focus: operate shared infrastructure, enable teams to use it, and establish company rules for that use.

### Connect the platform to the product experience

[24:42](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1482s)

The platform's internal focus leaves another coordination problem: how AI appears to customers across the product. Debois introduces the [unFIX Experience Crew](https://unfix.com/experience-crew) as a complementary organizational idea from an alternative to Team Topologies. Applied to AI, the crew works across feature teams to understand what they are building and make the resulting user experience consistent.

That adds an experience layer to the combination of CloudOps, SecOps, developer experience, the data platform, and AI infrastructure. Debois advises against this structure for a 10-person company and suggests considering it when scaling to 10 or more teams. It is a judgment about coordination needs, not a requirement to create departments before there is enough work to justify them.

The benefit comes from collaboration between specialties. SecOps brings access-control and governance expertise; CloudOps knows provisioning and cloud vendors. Bringing those capabilities into a shared area gives teams a better chance of solving the whole problem together.

### Who owns the guardrails?

[26:42](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1602s)

In the Q&A, Debois first agrees to share the slides. The substantive follow-up asks how guardrails should be delivered: should they sit in front of every served model and be documented on its model card, or should product teams be responsible for consuming a separate guardrail service correctly?

Debois describes layered ownership. Central governance supplies the generic rules; individual teams add rules appropriate to their use cases. The shared component avoids duplicating common policy, while self-service lets teams address requirements specific to their products. His answer establishes those responsibilities without prescribing a particular enforcement topology. The platform provides a common baseline, and product teams build their own protections on top.

## Resources

- [DevOps Topologies](https://web.devopstopologies.com/) — The earlier organizational patterns that helped inform Team Topologies.
- [Team Interaction Modeling with Team Topologies](https://teamtopologies.com/key-concepts-content/team-interaction-modeling-with-team-topologies) — A practical guide to mapping team responsibilities and collaboration, service and facilitation interactions.
- [unFIX Experience Crew](https://unfix.com/experience-crew) — An organizational pattern for improving the customer journey across products, services and teams.
- [Ironies of Automation](https://tc.ifac-control.org/4/1/newsletter/ironies-of-automation/view) — IFAC's access page for Lisanne Bainbridge's foundational 1983 paper on human operators and automation.
- [Ironies of Generative AI](https://arxiv.org/abs/2402.11364) — Research connecting generative AI's evaluation burden and workflow disruptions with earlier human-factors findings.
- [What is it like to program with artificial intelligence?](https://advait.org/publications-web/sarkar-2022-programming-ai/) — Research on AI-assisted programming, including code checking, debugging, developer experience and Copilot acceptance patterns.

## Slides

### Introducing Org & Tech Change

A pilot team tries something new, then two or three more teams repeat it. Lessons and patterns help scale the change across all teams as maturity grows.

[2:01](https://www.youtube.com/watch?v=5qNXdLbEdew&t=121s) [Diagram showing organizational change progressing through incubate, repeat, and scale as maturity increases.](https://assets.ai.engineer/sites/aiecode2025/sha256/8dd476ef18104abae5a86a8ebd019f70afcfcf54101e69aaa3931e8760997cb4.webp)

Introducing Org & Tech Change

Incubate
Repeat
Scale

Maturity →

AIE
Microsoft
smol ai

### Change Agents want an identity

AI Engineer names a role at the friction between data science and engineering. Like DevOps, the label is ambiguous, but it gives people a way to find emerging stories while the field evolves.

[2:51](https://www.youtube.com/watch?v=5qNXdLbEdew&t=171s) [Data science and engineering appear alongside AI Engineer, DevOps, Agile, and ITIL; text discusses ambiguous names and labels for sharing stories.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a01a60a811b810b1edfc187ea445d7b4b96a127efdb7c50c918ac4c926bbb81.webp)

Change Agents want an identity

Data science
Engineering

AI Engineer
DevOps
Agile
ITIL

Names are ambiguous
Yet resonate with a pain
Have a matching technology shift
Act a label to get the stories out

AIE
Microsoft
smol ai

[2:56](https://www.youtube.com/watch?v=5qNXdLbEdew&t=176s) [Data science and engineering are shown with friction between them. AI Engineer is listed alongside DevOps, Agile, and ITIL; labels help people find emerging stories.](https://assets.ai.engineer/sites/aiecode2025/sha256/e133bdf9c5a8678a895806e111752b047a50f0d6ca7c6019c70d2d9dc120b80e.webp)

Change Agents want an identity

Data science
Engineering

AI Engineer
DevOps
Agile
ITIL

Names are ambiguous
Yet resonate with a pain
Have a matching technology shift
Act a label to get the stories out

AIE
Microsoft
smol ai

### Org shift - Shift Right - Breaking Silos

Engineers joined the data science team as GenAI work moved toward integration; the organization then spread the work across teams and into a platform. The role spectrum places AI Engineer between research and full-stack engineering.

[4:06](https://www.youtube.com/watch?v=5qNXdLbEdew&t=246s) [Diagrams show data science and engineering moving toward integrated teams and a platform, above a spectrum of roles from research to full-stack engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/4da0a106e563092d5f8379c939ca93796d0451444660f3490350772ac1b76dce.webp)

Org shift - Shift Right - Breaking Silos

Data science
Engineering
Data science
Data science
Engineering
Data science
Platform

Data/Research constrained
API
Product/User constrained

ML Researcher / Research Scientist
- training
- evals

ML Engineer/ Data Scientist/ Research Engineer
- inference
- data

AI Engineer
- chains/agents
- tooling & infra

Fullstack Engineer
- product
- platform

https://www.latent.space/p/ai-engineer

AIE
aws

[4:14](https://www.youtube.com/watch?v=5qNXdLbEdew&t=254s) [Diagram showing data science and engineering teams integrating around a platform, above a spectrum of roles from research to full-stack engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/57e98c34fbe26a2bb4300cd861d13c8e6c0be3efe5c06dff219ec09efef51a7d.webp)

Org shift - Shift Right - Breaking Silos

Data science
Engineering
Data science
Data science
Engineering
Data science
Platform

Data/Research constrained
API
Product/User constrained

ML Researcher / Research Scientist
- training
- evals

ML Engineer/ Data Scientist/ Research Engineer
- inference
- data

AI Engineer
- chains/agents
- tooling & infra

Fullstack Engineer
- product
- platform

https://www.latent.space/p/ai-engineer

AIE

### Steps to scale the change

The speaker wants application developers to scale GenAI within their teams, supported by a platform, enablement, and governance.

[7:26](https://www.youtube.com/watch?v=5qNXdLbEdew&t=446s) [Platform, enablement and governance are listed as steps to scale the change.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d3895b340d973c4f24e5ed97bb8953da001a25207c7d2667e110e6e05d7d8fe.webp)

Steps to scale the change

🚄 Platform
✨ Enablement
🚨 Governance

AIE
aws

### Platform

The speaker introduces services that go into an AI platform, starting with access to models appropriate for the company.

[7:47](https://www.youtube.com/watch?v=5qNXdLbEdew&t=467s) [Platform slide asks what services go into the platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d76a33a5468e811ef1fd80e0708182d6a6d3e67c254f8444a756be989fc209b.webp)

🚄 Platform

What services go into your platform

AIE
aws

### Unstructured data - Vector Databases

Vector search has spread from dedicated vector databases into traditional databases. The slide groups examples by product type and licensing category.

[8:36](https://www.youtube.com/watch?v=5qNXdLbEdew&t=516s) [Matrix comparing dedicated vector databases and databases supporting vector search by licensing category.](https://assets.ai.engineer/sites/aiecode2025/sha256/a89efd1c49d902692bce01bb454a48f24e2d6fc3cbfa2caaaf65cdc0d947a7ce.webp)

Unstructured data - Vector Databases

Dedicated vector databases
Databases that support vector search

Open source (Apache 2.0 or MIT license)
Dedicated vector databases: chroma, marqo, vespa, Qdrant, LanceDB, Milvus
Databases that support vector search: OpenSearch, ClickHouse, PostgreSQL, cassandra

[illegible]urce available or commercial
Dedicated vector databases: Weaviate, Pinecone
Databases that support vector search: elasticsearch, redis, ROCKSET, SingleStore

Interface text:
AIE
aws

### RAG as a Service

A platform team can connect company data sources and share those connectors across teams, so each team does not have to build them separately. The speaker notes that some call this RAGOps.

[8:55](https://www.youtube.com/watch?v=5qNXdLbEdew&t=535s) [Slide showing a data-source selection screen with cloud and on-premises connectors.](https://assets.ai.engineer/sites/aiecode2025/sha256/cabcdd32880326d7753e8f63015f5c5df66248337d7b92ba367b259ccf699c2b.webp)

RAG as a Service
RAGOPS ?

Interface text:
Data sources (0) Info
Select the data source that you want to configure. You can configure up to 5 data sources per application.

Most popular
Amazon S3 — Cloud
Web crawler
Upload files — Available after retriever is created

All | Cloud | On-prem
Search

Adobe Experience Manager — Cloud, On-prem
Alfresco — Cloud, On-prem
Amazon FSx (Windows) — Cloud
Amazon RDS (MYSQL) — Cloud
Amazon RDS (Microsoft SQL Server) — Cloud
Amazon RDS (Oracle) — Cloud
Amazon RDS (PostgreSQL) — Cloud
Aurora (MYSQL) — Cloud
Aurora (PostgreSQL) — Cloud

AIE
aws

[9:09](https://www.youtube.com/watch?v=5qNXdLbEdew&t=549s) [RAG as a Service slide showing a data-source connector selection screen and the question “RAGOPS?”](https://assets.ai.engineer/sites/aiecode2025/sha256/edcd91047b69e36761d9907697262fcabfec4fe2a630f3300106c78e80f15921.webp)

RAG as a Service
RAGOPS ?

Interface text:
Data sources (0) Info
Select the data source that you want to configure. You can configure up to 5 data sources per application.

Most popular
Amazon S3 — Cloud
Web crawler
Upload files — Available after retriever is created

All | Cloud | On-prem
Search

Adobe Experience Manager — Cloud, On-prem
Alfresco — Cloud, On-prem
Amazon FSx (Windows) — Cloud
Amazon RDS (MYSQL) — Cloud
Amazon RDS (Microsoft SQL Server) — Cloud
Amazon RDS (Oracle) — Cloud
Amazon RDS (PostgreSQL) — Cloud
Aurora (MYSQL) — Cloud
Aurora (PostgreSQL) — Cloud

AIE
Microsoft
smol ai

### Unified Model API Proxies and Access Control

LiteLLM routes requests to multiple model providers using the OpenAI format, with load balancing and fallbacks. The speaker also describes proxies controlling who can access models.

[10:04](https://www.youtube.com/watch?v=5qNXdLbEdew&t=604s) [Slide showing LiteLLM routing requests to multiple model providers, with load balancing and fallbacks.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f28c3fc6b5ba33367ad68da53e2fd56298440cc5d35c3b2ea17948ee44b5ef9.webp)

Unified Model API proxies - ACL

Load Balance across OpenAI

LiteLLM handles loadbalancing, fallbacks and spend tracking across 100+ LLMs. All in the OpenAI format

LiteLLM
cohere
PaLM 2

[illegible]hub.com/BerriAI/litellm
https://www.litellm.ai/

Interface text:
LiteLLM
Schedule Demo
LiteLLM Docs
Github 8.4K
Try LiteLLM Cloud Free
Deploy LiteLLM Open Source →
AIE
aws

### Prompt Debugging & Traceability

Capture production prompts in the observability layer: one prompt can lead to several iterations. The trace dashboard highlights a run with its latency and token count.

[10:37](https://www.youtube.com/watch?v=5qNXdLbEdew&t=637s) [A trace dashboard highlights an LLM run with latency and token information.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ada69e4e0b908785b1b45e61531c35823fd5b2071c531b9dcaa324b3c991e01.webp)

Prompt Debugging & Traceability

[illegible]tps://cobusgreyling.medium.com/langsmith-1dd01049c3fb

AIE
Microsoft
smol ai

Interface text:
Basic_Project_1
P50 Latency: 0.57s
P99 Latency: 24.77s
Total Tokens: 441
Traces  All Runs  Setup
eg. eq(run_type, "chain")
Columns
Run Type | Name | Input | Start Time | Latency | Tokens | Tags
LLM | ChatOpenAI | human: What is the ... | 01/08/2023, 09:37:43 | 7.29s | 99
Chain | LLMChain | What is the year of ... | 31/07/2023, 20:50:42 | 0.08s | 56
Chain | LLMChain | What is the year of ... | 31/07/2023, 20:50:23 | 0.07s | 56
Chain | LLMChain | What is the year of ... | 31/07/2023, 20:49:46 | 0.37s | 56
Chain | LLMChain | What is the year of ... | 31/07/2023, 20:23:44 | 0.77s | 56
Chain | LLMChain | What is the square r... | 31/07/2023, 20:22:44 | 26.09s | 58
Chain | LLMChain | Who won the FIFA ... | 31/07/2023, 20:06:18 | 0.37s | 40
ChatOpenAI | human: Hello, world! | 31/07/2023, 16:43:37 | 1.13s | 20
Rows per page: 10
1–8 of 8
Filters
Full-Text Search
Search...
Name
LLMChain 6
ChatOpenAI 2
Run Type
Chain 6
Llm 2
Status
Success 8
Other
Latency >= 10s
Tokens >= 1,000
Today
Canceled

### Enablement

Enablement means guiding teams to use the platform’s tools and gathering feedback on what works and what does not.

[12:17](https://www.youtube.com/watch?v=5qNXdLbEdew&t=737s) [Enablement slide asks how to help customers of the platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/454fc86aec764b356101a7fc7ac2e9ef0d709226e97bcd079c3acfb9d5db957a.webp)

✨ Enablement

How can we help the customers of our platform ?

AIE
aws

### Encourage Experimentation / Prototyping

The Azure chat playground illustrates prototyping with different models and connected data. Include product owners in experimentation and connect company data securely.

[12:44](https://www.youtube.com/watch?v=5qNXdLbEdew&t=764s) [Azure chat playground screenshot highlights different models, connected data, and inclusion of product owners.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa75f2f842bf7464fdfe13aa424543cf07ebf3496cece7a51c3482ae9133249f.webp)

Encourage Experimentation / Prototyping

Different Models
Connected with your Data
Including Product Owners
https://azure.microsoft.com/en-us/products/ai-studio

AIE

Interface text:
Azure OpenAI
Chat playground
Playground
Chat
Completions
DALL-E (Preview)
Management
Deployments
Models
Data files
Quotas
Content filters (Preview)
Deploy to
Import setup
Export setup
Show panels
Replay chat
Clear chat
Playground Settings
View code
Show JSON
Assistant setup
Prompt
Add your data (preview)
Apply changes
Use a system message template
Using templates
Use a template to get started, or just start writing your own system message below. Want some tips? Learn more
Select a template
System message
[illegible] find information.
Start chatting
Test your assistant by sending queries below. Then adjust your assistant setup to improve the assistant's responses.
Configuration
Deployment
Parameters
Deployment *
gpt4-pdb
Session settings
Past messages included
10
Current token count
Input tokens progress indicator
11/32768

[12:58](https://www.youtube.com/watch?v=5qNXdLbEdew&t=778s) [Azure AI chat playground illustrating experimentation with models and company data, including product owners.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f04bf77e7b7136fa898372abc020448d893994e8075ab771e08053875b4505e.webp)

Encourage Experimentation / Prototyping

Different Models
Connected with your Data
Including Product Owners
https://azure.microsoft.com/en-us/products/ai-studio

AIE
Microsoft
smol ai

Interface text:
Azure OpenAI
Chat playground
Playground
Chat
Completions
DALL-E (Preview)
Management
Deployments
Models
Data files
Quotas
Content filters (Preview)
Deploy to
Import setup
Export setup
Show panels
Replay chat
Clear chat
Playground Settings
View code
Show JSON
Assistant setup
Prompt
Add your data (preview)
Apply changes
Use a system message template
Using templates
Use a template to get started, or just start writing your own system message below. Want some tips? Learn more
Select a template
System message
[illegible] find information.
Start chatting
Test your assistant by sending queries below. Then adjust your assistant setup to improve the assistant's responses.
Configuration
Deployment
Parameters
Deployment *
gpt4-pdb
Session settings
Past messages included
10
Current token count
Input tokens progress indicator
11/32768

### Learn from Orchestration Frameworks

LangChain and Guidance are examples of frameworks the speaker recommends for learning how orchestration works and for education and documentation.

[13:11](https://www.youtube.com/watch?v=5qNXdLbEdew&t=791s) [Slide showing LangChain, Guidance, LlamaIndex, Griptape, and DSPy; the presenter inset partly covers the LlamaIndex link.](https://assets.ai.engineer/sites/aiecode2025/sha256/a07b540a0dc331f697d8c760691fe20bcf13bedfaaf0e3554879a157c45d766a.webp)

Learn from Orchestration Frameworks

LangChain
https://www.langchain.com/

{{guidance}}
https://github.com/guidance-ai/guidance

[illegible].llamaindex.ai/

griptape
https://www.griptape.ai/

DSPy
https://github.com/stanfordnlp/dspy

AIE
Microsoft
smol ai

### GenAI Project Pitfalls

Start with a real business case before optimizing costs or focusing heavily on model training. End-user feedback also needs an owner who can interpret it in the product’s domain.

[14:07](https://www.youtube.com/watch?v=5qNXdLbEdew&t=847s) [Four pitfalls: no real business use case, excessive focus on model training, premature cost optimization, and feedback without ownership.](https://assets.ai.engineer/sites/aiecode2025/sha256/d71df3b7ef911f4231ba11a2d6f178e11477de33fc08bf6ccc1bb981f55a62a1.webp)

GenAI Project Pitfalls 💥

• 💡 No real business use case - Avoid AI for the sake of AI
• 🧪 Too focused on model training & fine tuning - Needs focus on customer
• 💸 Premature cost optimization - it will only get cheaper
• 👂 Enduser feedback not actionable - Needs ownership

AIE
@patrickdebois

[14:17](https://www.youtube.com/watch?v=5qNXdLbEdew&t=857s) [Slide lists pitfalls involving business use cases, model training, premature cost optimization, and ownership of user feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b052b795bf491c20144c47c203ff99ae9c24321305a2d38d9083e5feb1ef67f.webp)

GenAI Project Pitfalls 💥

• 💡 No real business use case - Avoid AI for the sake of AI
• 🧪 Too focused on model training & fine tuning - Needs focus on customer
• 💸 Premature cost optimization - it will only get cheaper
• 👂 Enduser feedback not actionable - Needs ownership

AIE
@patrickdebois
Microsoft
smol ai

### Test Mindset - Evaluation types

Evaluation combines simple text checks, model-based judgments, semantic distance, and human feedback. De Bois questions the circularity of using an LLM to judge another LLM.

[17:43](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1063s) [Table compares exact testing, pattern matching, model checks, semantic distance, LLM evaluation, and human feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7180bb12952ef842995340fd5eed05c3abaf937af7137cdaa5f8a3e0d95ca43.webp)

Test Mindset - Evaluation types

Exact Testing
Is this text > 20 characters

Pattern Matching
regex(/keyword/)

Ask a Model
Sentiment, Quality, Toxic, ...

Semantic Distance
Related concepts ?

Ask an LLM
Do you think this is a toxic answer ?

Ask a Human
Got any Feedback?

@patrickdebois

AIE
Microsoft
smol ai

[17:48](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1068s) [Table compares exact testing, pattern matching, model-based checks, semantic distance, LLM evaluation, and human feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/39523a1d736c31a244fa2227fded57ed8c861f94d613ebb70c2747f99b53fe47.webp)

Test Mindset - Evaluation types

Exact Testing
Is this text > 20 characters

Pattern Matching
regex(/keyword/)

Ask a Model
Sentiment, Quality, Toxic, ...

Semantic Distance
Related concepts ?

Ask an LLM
Do you think this is a toxic answer ?

Ask a Human
Got any Feedback?

@patrickdebois

AIE
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=5qNXdLbEdew&t=0s) [upbeat music] Thank you. Especially after lunch, that applause tell, you know, is, tells double, so.

[0:19](https://www.youtube.com/watch?v=5qNXdLbEdew&t=19s) So yeah, you need a platform team. Um, so usually I'd skip over this slide, but this time I'm gonna explain you my background because it's important to understand what the next things is that I'm completely biased in this, what I'm gonna talk about.

[0:35](https://www.youtube.com/watch?v=5qNXdLbEdew&t=35s) So why am I biased? In 2009, I organized a small conference called DevOpsDays. There were 60 people,

[0:44](https://www.youtube.com/watch?v=5qNXdLbEdew&t=44s) and this is how the world kind of got into DevOps as a word. So my position has been privileged to be there from the beginning and seen 15 years of a new kind of phenomenon, discussion, transformation taking root across the whole enterprise.

[1:02](https://www.youtube.com/watch?v=5qNXdLbEdew&t=62s) So that's kinda like why a lot of what I'm talking about is things from the DevOps space, what we learned here, and I'm trying to apply this to whatever is the new thing we're doing.

[1:13](https://www.youtube.com/watch?v=5qNXdLbEdew&t=73s) Um, when you apply your old paradigms to the new world, it could either be correct, and it could be totally wrong. So I'm explaining what I think that I'm seeing.

[1:26](https://www.youtube.com/watch?v=5qNXdLbEdew&t=86s) Um, and one of the particular interest is after seeing DevSecOps and kind of whatever ops in the world, I especially like this part of the GenAI, uh, in particular for that it's not...

[1:40](https://www.youtube.com/watch?v=5qNXdLbEdew&t=100s) Like, for me, it's automation intelligence. So a lot of my work on DevOps was automating pipeline, making them more robust, and kind of making sure we're delivering the value on this.

[1:51](https://www.youtube.com/watch?v=5qNXdLbEdew&t=111s) So that's why kind of I, I get excited about this new world.

[1:57](https://www.youtube.com/watch?v=5qNXdLbEdew&t=117s) So over the years, I have [REDACTED:physical_attribute], right? So I've seen my fair share of new things coming in the industry, and the pattern is usually

[2:10](https://www.youtube.com/watch?v=5qNXdLbEdew&t=130s) you have one team that's the, the pilot team, and something new happens. We're gonna try this out. Then you scale this out into two or three more teams doing this, and then you extrapolate some learning, some patterns, and eventually you wanna scale this out to all the teams.

[2:28](https://www.youtube.com/watch?v=5qNXdLbEdew&t=148s) This is what happened with cloud, the infrastructure. You know, initially it was a small thing, uh, but then eventually this led into an abstraction layer of us kind of making this easier for other teams so they don't have to understand the whole space, but they can move faster within their domain.

[2:50](https://www.youtube.com/watch?v=5qNXdLbEdew&t=170s) Like DevOps, everybody was saying DevOps is a bad name. W- we kind of hate it, but in the end, the industry stuck with us. AI engineer is a bad name.

[3:03](https://www.youtube.com/watch?v=5qNXdLbEdew&t=183s) What does it mean? Nobody knows. But it's also the advantage that you can bring everything to the table and let it grow. So if you make it too defined, then we are not evolving that way.

[3:16](https://www.youtube.com/watch?v=5qNXdLbEdew&t=196s) That... So that, that, that has kind of evolved in... The first there was the agile team. They were, like, really proud as the change agent, happened to ITIL, and we're kind of bringing this in.

[3:28](https://www.youtube.com/watch?v=5qNXdLbEdew&t=208s) I think what I learned over the years of DevOps is the name does not matter that much. In the beginning, it's really important that you have a label that you can search and find all the emerging stories of people doing this thing.

[3:42](https://www.youtube.com/watch?v=5qNXdLbEdew&t=222s) So that's how I look at AI engineer as a term. If you search for this, it's gonna be on your job post. It's gonna be on your website. This is likely the term that kind of fits into f- us finding each other into the new space.

[3:59](https://www.youtube.com/watch?v=5qNXdLbEdew&t=239s) And then with DevOps, we're trying to bridge something in the organization. Two things that had friction, that did not work together. And I've witnessed firsthand in the company after, you know, now two years building GenAI application, the friction was in the beginning.

[4:17](https://www.youtube.com/watch?v=5qNXdLbEdew&t=257s) We all wanna do this GenAI thing. It hit first the data science team, and then they were, like, screaming, "Hang on. We're not used to running things in production."

[4:27](https://www.youtube.com/watch?v=5qNXdLbEdew&t=267s) Right? So all of a sudden you got friction. They know their world. We know their o- the other world, and we started slowly moving some engineers into the data science team.

[4:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=278s) Eventually, the data science team became smaller of that part because it was less about the data science. We heard it this morning. It was more about integration thing, and the ratio of the data science got smaller compared to the engineering.

[4:51](https://www.youtube.com/watch?v=5qNXdLbEdew&t=291s) And then what we did in the organization is we started scaling this out to the different teams, and eventually moved this into the platform. So that kind of friction of the movement in an organization is something I, uh, you know, that, that I see happen a lot.

[5:07](https://www.youtube.com/watch?v=5qNXdLbEdew&t=307s) The data was left on their own in their own data lake, and then now we're actually binding and bonding back together in that way. So in that way, that's the shift right.

[5:16](https://www.youtube.com/watch?v=5qNXdLbEdew&t=316s) Security was shift left, so we're always shifting around, like whatever the complexities we can do.

[5:25](https://www.youtube.com/watch?v=5qNXdLbEdew&t=325s) Maybe first a question. Who is familiar with team topologies as a concept?

[5:29](https://www.youtube.com/watch?v=5qNXdLbEdew&t=329s) Wow. Okay. Not that many. So first you had DevOps as a concept, bringing two teams back in the same room, having them collaborate. That was one way of collaborating.

[5:43](https://www.youtube.com/watch?v=5qNXdLbEdew&t=343s) But that did not scale across the org. So slowly, uh, they became their own team that abstracted pieces away, the Kubernetes pipeline, all the infrastructure that, uh, developers needed to build, the security tooling.

[5:59](https://www.youtube.com/watch?v=5qNXdLbEdew&t=359s) That kind of moved into a platform team. Team Topologies originally was actually called DevOps Topologies because it was a m- like a movement of the teams, uh, reacting to the you build it, you run it.

[6:13](https://www.youtube.com/watch?v=5qNXdLbEdew&t=373s) So we-- eventually, the teams you-- that are building and running it, we call them the, uh, kind of the feature teams, and they run on top of the platform.

[6:24](https://www.youtube.com/watch?v=5qNXdLbEdew&t=384s) And the platform c- team can work together with those teams either by just giving it an API, right? Run as a service, and you're done. We can do more collaboration to see, hey, you know, what are you building?

[6:39](https://www.youtube.com/watch?v=5qNXdLbEdew&t=399s) Let's build this, make sure that the platform is helping you. So it's a little more like they're our customers, we're building a product, you're building on top of that.

[6:48](https://www.youtube.com/watch?v=5qNXdLbEdew&t=408s) And then we can do facilitation if they're having some issues, and we're helping them. So kind of that interaction team. Uh, there's a whole book on this, Team Topologies, and it helps people organize kind of their infrastructure teams, their platform teams, their SRE teams.

[7:02](https://www.youtube.com/watch?v=5qNXdLbEdew&t=422s) So it's a known pattern how we deal with these kind of abstractions of new te-technology, bringing it into the company.

[7:10](https://www.youtube.com/watch?v=5qNXdLbEdew&t=430s) So what I'm explaining here is this whole GenAI thing, the impact is gonna be like not just the data science team doing this. But what I tried to do in the last two years is bring this to the traditional application developers, so they can scale this out into their own teams.

[7:30](https://www.youtube.com/watch?v=5qNXdLbEdew&t=450s) We heard it this morning. What's the ideal team? It's the application developers, the traditional ones, and some of the data science on top of that. But there's a few hurdles to coming to that.

[7:41](https://www.youtube.com/watch?v=5qNXdLbEdew&t=461s) First piece that I'm talking about is the platform.

[7:45](https://www.youtube.com/watch?v=5qNXdLbEdew&t=465s) So what services go into an AI platform? And, you know, you're here at the fair. There's like, I don't know, how many vendors. There's lots of hardware. So I have no affiliation with any of the things that I'm showing.

[7:57](https://www.youtube.com/watch?v=5qNXdLbEdew&t=477s) I'm just gonna show you some pieces that go into that platform that you're providing.

[8:03](https://www.youtube.com/watch?v=5qNXdLbEdew&t=483s) First piece, access to models, right? It's this first thing they can do. Instead of every team going out to figure out what's the best model, where do you get access, that team kind of figures out what is appropriate for our company.

[8:16](https://www.youtube.com/watch?v=5qNXdLbEdew&t=496s) You pick your poison, what you like, what's your vendor, typically related to your cloud vendor. So there's a relationship with the CloudOps team and kind of bringing this in.

[8:26](https://www.youtube.com/watch?v=5qNXdLbEdew&t=506s) Next piece that you heard, like RAG, let's have a vector database. In the beginning, you had like very specific ones. Now you see they're all getting inside of the traditional vendors of kind of the databases and whatever.

[8:39](https://www.youtube.com/watch?v=5qNXdLbEdew&t=519s) So it's not that special anymore, but you need to understand what it is, what is a vector, what is embeddings, how we're helping this. So that's one of the other pieces you bring when you're having the RAG in.

[8:51](https://www.youtube.com/watch?v=5qNXdLbEdew&t=531s) And having that ve-- uh, vector database is not enough, so you bring in connectors to all your data sources. So that's another piece that that team builds in the infrastructure.

[9:02](https://www.youtube.com/watch?v=5qNXdLbEdew&t=542s) They connect whatever is out there in your company, and they expose that to all the other teams instead of hem-- them having to do this all the time one by one on teams and figuring out how that works.

[9:14](https://www.youtube.com/watch?v=5qNXdLbEdew&t=554s) Some have called this RAGOps, you know, whatever ops, right? There's always something to be, uh, managing and running on that.

[9:21](https://www.youtube.com/watch?v=5qNXdLbEdew&t=561s) And then we had version control for code. Obviously, we wanna have version control for models, so that's another piece they can provide. So instead of them figuring out on their own how we do this, there's a centralized repository.

[9:33](https://www.youtube.com/watch?v=5qNXdLbEdew&t=573s) Other teams can reuse, uh, some of the registry-- uh, some of the models. So it becomes like visible, like a library to all the other pieces as well.

[9:44](https://www.youtube.com/watch?v=5qNXdLbEdew&t=584s) And then especially the larger enterprises, uh, it's almost like they wanna be cloud agnostic. They wanna be model provider agnostic. So I'm not saying, like in storage, we used to have S3 as being the standard protocol.

[9:58](https://www.youtube.com/watch?v=5qNXdLbEdew&t=598s) Now maybe that's the OpenAI protocol that being standardized on, like the one winner kind of takes it all. But it's also kind of building this in the access control, who gets access to what.

[10:10](https://www.youtube.com/watch?v=5qNXdLbEdew&t=610s) So there's like a proxy that these kind of environments are building, uh, so not everybody can just go out and use whatever they want. So there's a little bit of that.

[10:21](https://www.youtube.com/watch?v=5qNXdLbEdew&t=621s) Then when you've been running this, you want to have similar to your observability and tracing, you wanna capture whatever prompts is running in production. Uh, so that observability layer, you wanna enhance your existing observability layer, but it's a little bit different.

[10:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=638s) Uh, you know, one prompt, it's rarely one prompt. Usually one prompt leads to five, six iterations, questions. So that's another piece of this, and you don't want every team to figure this out on their own.

[10:49](https://www.youtube.com/watch?v=5qNXdLbEdew&t=649s) And then when you run it in production, you want to have monitoring on your data quality. And I know like, you know, LMM-- MLOps had this before, but this is kind of a new thing that the traditional monitoring and metrics providers are not very capable of because like you had a health check for your API calls, now

[11:08](https://www.youtube.com/watch?v=5qNXdLbEdew&t=668s) you have a health check for evals running all the time in production. So you will notice if the model has changed. You will notice if the end user is doing strange things.

[11:17](https://www.youtube.com/watch?v=5qNXdLbEdew&t=677s) You notice-- so kind of that observability kind of is built in.

[11:22](https://www.youtube.com/watch?v=5qNXdLbEdew&t=682s) And then lastly, you know, there's, there's many things like caching services, and you go on, like feedback as a service being also discussed this morning. Uh, not just thumbs up, thumbs down, try again, but also kind of in, uh, inline editing of the solution be-because you get better feedback.

[11:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=698s) And again, you don't want every team to build this service because it's quite expensive, uh, and you want this to be centrally, uh, managed in a good way.

[11:47](https://www.youtube.com/watch?v=5qNXdLbEdew&t=707s) So these are just a few pieces, uh, to bring it on the board. Um, if you ever seen the Kubernetes ecosystem slide, this one is also expanding rapidly, right?

[11:58](https://www.youtube.com/watch?v=5qNXdLbEdew&t=718s) So there's similarities on whatever hardware we're running. It's just gonna keep growing, uh, whatever solution we have out there. So that gives you a little bit of an idea is it's not just clouds and APIs, but this is a new set of infrastructure that you're running.

[12:15](https://www.youtube.com/watch?v=5qNXdLbEdew&t=735s) So next piece is you have all the infrastructure, and you provide it to the teams. And one of the things we learned is that it's not just enough to say, "Here is a bunch of things you could use."

[12:25](https://www.youtube.com/watch?v=5qNXdLbEdew&t=745s) Like any good company, you guide your customers to use them. That's the enablement to kind of whatever you're providing that they're able to use. This is also the place where you get feedback, what is working and not working from your team.

[12:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=758s) So what does the enablement look like for that team?

[12:42](https://www.youtube.com/watch?v=5qNXdLbEdew&t=762s) You provide prototyping tools for them to easier do experimentation. And it's been mentioned before, don't forget the product owners. They will learn. They wanna experiment. So it's a simple thing you do to kind of get them excited, having them playing around, find the right use case, uh, of their stuff.

[13:00](https://www.youtube.com/watch?v=5qNXdLbEdew&t=780s) And then you also connect it with the data of your company. So you do that in a secure way so they can experiment whatever data is there.

[13:09](https://www.youtube.com/watch?v=5qNXdLbEdew&t=789s) And frameworks are great, and I'll come back to that later, for learning. How does it work? What can we all do? And I learned a ton through these, uh, things, and there is a flavor for everybody, like large data, kind of like you like more like Microsoft stack.

[13:24](https://www.youtube.com/watch?v=5qNXdLbEdew&t=804s) Definitely framework is, uh, one of the things, uh, to give them and hand them over, uh, to learn from. And it also helps in education, documentation, uh, going from there.

[13:37](https://www.youtube.com/watch?v=5qNXdLbEdew&t=817s) They want a local dev environment. They wanna feel safe to do some coding, experimentation when they travel, kind of like that they don't wanna have the resources. They're very, like, focused on having this local stack of development, uh, when working on this.

[13:52](https://www.youtube.com/watch?v=5qNXdLbEdew&t=832s) There's pro and cons, but, you know, to get them excited, it's one of the things that you can definitely do. And we've seen this morning you can run more and more of these quality models on your laptop, so that kind of helps them in their faster iteration.

[14:05](https://www.youtube.com/watch?v=5qNXdLbEdew&t=845s) So things that I saw that actually go a lot bad is what is the actual use case? Uh, I've seen companies shout like, you know, we need to have the GenAI.

[14:17](https://www.youtube.com/watch?v=5qNXdLbEdew&t=857s) Every part of the product needs to have GenAI. But we're in the phase that we're still figuring out the real use case for a lot of things. And we're...

[14:26](https://www.youtube.com/watch?v=5qNXdLbEdew&t=866s) I, I'm not gonna say we're running on the marketing budget, but sometimes it's a little bit like that, right? And another pitfall is if you're very focused on kind of model training and fine-tuning, I can tell you that companies run by the data science, and it might take a while until we actually get something in production as

[14:45](https://www.youtube.com/watch?v=5qNXdLbEdew&t=885s) well. Um, an over-focus on, uh, kind of,

[14:51](https://www.youtube.com/watch?v=5qNXdLbEdew&t=891s) uh, cost, like run local, not use the perfect model. There's a lot of cost that, you know, goes down the drain that way. So we know that the models are getting cheaper.

[15:01](https://www.youtube.com/watch?v=5qNXdLbEdew&t=901s) So focus on the first thing, get that business case focused, and the rest we'll kind of like, we'll, we'll sort out later. If there's actually ROI, we'll reduce the cost on that.

[15:09](https://www.youtube.com/watch?v=5qNXdLbEdew&t=909s) And then end user feedback, um, the engineers often don't know what to do with the feedback because the product owner is the only one that knows the domain. It was mentioned this morning.

[15:21](https://www.youtube.com/watch?v=5qNXdLbEdew&t=921s) So that's kind of also like, yeah, we get all the feedback, but now what? Like, yeah, we're not used to handling this kind of feedback. We're used to handling error stacks as well.

[15:30](https://www.youtube.com/watch?v=5qNXdLbEdew&t=930s) So those are a few of the observations.

[15:34](https://www.youtube.com/watch?v=5qNXdLbEdew&t=934s) And that brings me to there's a lot of emphasis on developer experience in general and improving the productivity. But what does it take to be building GenAI applications? What kind of pains do you have?

[15:49](https://www.youtube.com/watch?v=5qNXdLbEdew&t=949s) Like, I've yet to see the developer that knows what to pick as a model. They just pick one, and it's r- it's okay, but you gotta start somewhere. Access to the data in the test is hard.

[16:02](https://www.youtube.com/watch?v=5qNXdLbEdew&t=962s) Uh, I hear a lot of stories that people use the framework. They were bitten by a framework. They don't like the framework anymore because the frameworks were moving too fast.

[16:11](https://www.youtube.com/watch?v=5qNXdLbEdew&t=971s) Uh, but... And they start doing things themselves at the lower layer or APIs. I hope that's not the way we're going. I hope we eventually have an ecosystem of tools we're building on and not doing the DIY.

[16:23](https://www.youtube.com/watch?v=5qNXdLbEdew&t=983s) But it's kind of what I've seen in people in the journey. They get excited with the framework, and then it's like, "Okay, we, we got this. It's just a prompt, right?

[16:30](https://www.youtube.com/watch?v=5qNXdLbEdew&t=990s) Like, it's not that difficult." And in essence, they're right, but think about, like, the tracing, the monitoring, the observability, all the ecosystems you wanna kind of bring together.

[16:40](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1000s) And we went through, I think, eight different models over two years for LLMs to get, like, improvement of quality. And me looking as a VP engineering, I was like, "Hang on."

[16:57](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1017s) So every time we change the model, we have to rewrite and rework all the application prompts to make sure it works. That's very costly. And if it wasn't costly enough, they were doing this manually. [laughs]

[17:10](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1030s) So I had no guarantee that was actually gonna work. So we've talked enough about evals and working with evals, but it's definitely one of the pain points of you need to have testing evals before you do this refactoring.

[17:24](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1044s) And this brings me to the testing, which is every time that the engineers go like, "What?" Like, "How do I, like, write test for this?" Like it's, you know...

[17:33](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1053s) This morning we're like, "Oh, it's a vibe check. Okay, you know, I, I don't get it." It's like, that's not engineering rigor. So

[17:41](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1061s) I'm gonna give you, uh, like the buildup that they're saying. Like, "Well, I can do exact testing. That, that's easy, right? Just does it need to be twenty characters or regex?

[17:49](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1069s) I know that stuff. Sentiment check? Okay." Well, you know, all of a sudden you have to run a model, a helper model, uh, to bring that in. We can see a simple thing like similar to the semantic, uh, distance.

[18:04](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1084s) Is the question related to answer? That's an easy one. But then you're telling me, how do you check an LLM? Use another LLM?

[18:13](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1093s) Right? So if the AI doesn't work, just use more AI. Like, it blows my mind. Like, it's like a vicious circle, and it hasn't been really solved. But hey, it's the best thing we got, and I got it.

[18:23](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1103s) Like, and, and eventually there's a human feedback. So it's, it's a combination of a, a bit of everything there.

[18:30](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1110s) Um, so now I'm switching gears a bit. Like-

[18:35](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1115s) When you wanna bring this to the engineers, a lot of the engineers are afraid of the AI in the beginning. So how do you get them excited? And the usu- usual trick is we get them something on their coding copilot, because then they see kind of the productivity that this brings.

[18:52](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1132s) And so it goes hand in hand. Like, it's a little bit weird that the same team that is building the GenAI models is also being the evangelizing of kinda, like, take away the fear of developers using more AI.

[19:05](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1145s) And this one, uh, I know there's a lot of talk about, like, getting them more productive, but what I wanna show here is, yes, the copilot introduced a lot more code.

[19:18](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1158s) It was a lot more things to do. But review times went up, and the PRs sent by the actual AI were bigger, right? So we got faster, but then we started slowing down again.

[19:30](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1170s) So there's like... I'm not saying it's, like, a hundred percent compensating the other part, but there's definitely... It, it's like there's friction on the line.

[19:40](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1180s) And this brings me to a paper that where-- was very instrumental in the beginning of, uh, DevOps. It was called The Ironies of Automation. And now there's a new paper, The Ironies of GenAI Automation.

[19:52](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1192s) And so you see, um, the role of the person using any of the AI is changing from the person producing to somebody who's managing and reviewing things, right? So...

[20:05](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1205s) And you could say, "Well, you know, that, that's good." Like, you know. But you see you're spending more time in the review and kind of faster of the generation.

[20:13](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1213s) But it can only also lead to, "I don't understand anymore what this, this is doing because I don't have the domain model. I don't have the experience anymore." And then I'm just gonna say accept.

[20:26](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1226s) There's a funny story about GitHub Copilot being used, uh, and having a higher acceptance rate for suggestions in the weekend because [laughs] the developers are like, "Whatever." [laughs]

[20:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1238s) So... But we, we've seen this before, like, the automation, that acceptance, so we kind of have to go a-against this. And you would gradually, when you're not doing the producing job anymore, you'll lead, uh, like, lose situational awareness of when things, uh, go, uh, failing.

[20:56](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1256s) And this brings me to this slide. So

[21:00](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1260s) DevOps automating things. Go away. I'll re- I'll replace you with a shell script. That was kind of, you know, the left part, twenty percent automate. The other eighty percent that grew into the industry was preparing for failure.

[21:15](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1275s) First we had CI/CD. We created more tests, evals. Then we had monitoring because we needed to know what was going on in production. And then all of a sudden it's like, "Yeah, but if it goes down, like, can we prevent the failure?

[21:27](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1287s) We're gonna assume failure." [laughs] That was the big model shift. So we started design for failure, and then we needed to predict. But we didn't know what was going on.

[21:37](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1297s) But we need situational awareness. That was the whole observability. And then in the end we brought in chaos engineering to kind of inject failures so we can keep training when failures happened, right?

[21:48](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1308s) So you see kinda this paradigm of, yeah, we won a lot, but kind of that spun up a whole new thing of doing. I'm not saying everybody's doing this.

[21:58](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1318s) You can gladly skip all the tests if that's your risk appetite, but that's c- kind of the shift that, uh, it was going to. And it's also reducing cognitive load in when failure happening, uh, and kind of work from there.

[22:12](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1332s) So that's, you know, a little piece on enablement. It's not just, you know, kind of making them GenAI. It's also putting them at ease. And then there's the governance piece.

[22:21](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1341s) Uh, make a personal awareness program, right? Don't just copy things in. Uh, this is one that records your screen and looks everything on your screen, right? That, that's... It's getting scary in that way of leaking things.

[22:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1358s) Uh, but it also promises productivity, so it's like a-- it's a tension that you have to overcome in your company. Uh, you wanna have them opt out on training.

[22:47](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1367s) It's-- We've talked about this this morning. It's not always that clear where you opt out, when you opt out, uh, how to work from them. So you have to make, make them aware whenever they're putting a new service in there.

[22:58](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1378s) You wanna have them check the license, but that's overcome by just restricting the number of models you put in. Uh, it's not just all because it has the word open that it's open, right?

[23:07](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1387s) So we all learned that. Um, look at what the origin was for the model. Learn from that as well. So those are things that typically goes into a governance, uh, uh, workflow, um, of approval.

[23:22](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1402s) And then there's the [REDACTED:origin]... I come from Belgium, so risk levels assessment. What kind of workflow are you doing? Is it allowed? Should we do this? Kind of make this awareness a little bit of the law.

[23:31](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1411s) I'm not making them specialist lawyers, but at least, you know, they kind of have to know the basics of that as well.

[23:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1418s) And then there's prompt injection, but we learned it's not a solved problem. But you gotta have something, right, at least in place, uh, for the failures. Um, and then guardrails, uh, but I don't know if you ever done, like, intrusion detection alerting or web application firewalls.

[23:54](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1434s) You put all the rules and after a while you say, like, the logs are so big, you're like, "Whatever." So it's, it's not always a problem, but we, we kind of tell ourselves that there's guardrails in place, uh, uh, in there.

[24:06](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1446s) So let's hope we are, like, on the path of improving those kind of tools eventually, like, for preventing failure and optimizing for failure. And then we wanna, like, focus on the PII that goes over the wire, the metrics, the alerting, because we're now sending a lot more text over the wire, k- uh, kind of uncontrolled things, so

[24:26](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1466s) it needs a lot more scrutiny on there. So simple thing for me. These are the steps you bring in. You bring your infrastructure on one team. Uh, you kind of do the enablement on top of that.

[24:38](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1478s) And they also set the, the rules for your company to use this.

[24:42](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1482s) And I briefly talked about the platform team, which provides the internal focus and the internal enablement to the company. But, um, there's another model called the unFIX model, and it's a little bit like alternative to the team topologies.

[25:00](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1500s) And they also talk about the experience crew. And what the experience crew is, they make AI look consistent in the product. So they're a team that goes on to the feature teams.

[25:12](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1512s) It's like, "So what are you doing? How does it look like in the product? What's the UX?" And kind of they do this. They have a whole lot of number of crews, but it...

[25:21](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1521s) This is also used in companies and they call it, you know, the experience, the AI experience team that works kind of hand-in-hand. And so that leads me to CloudOps, SecOps, developer experience, put the data platform in there, put the AI platform infrastructure in there, and on top of that, do the AI, uh, kind of experiences in there.

[25:39](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1539s) Um, I would not recommend to do this for a 10-person company. Uh, don't do premature optimization. But if you're scaling out to 10 and more teams, that's probably a pattern that is known.

[25:52](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1552s) And the nice thing is you have cross-collaboration. Uh, the SecOps are really good at access control and governance, so they help each other. The CloudOps know what to spin up and know the cloud vendors.

[26:04](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1564s) And so there's a lot of kind of working together. And if you bring that into one area, you have a better chance of pulling this in, uh, and being like a collaboration across.

[26:15](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1575s) So that was me. And, you know, you can scan, scan the link and connect and, um, yeah, I don't know. What do you think? Wanna have a platform team for this, or are some people doing this already?

[26:27](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1587s) Uh, let me know. So thank you. [audience applauding] Questions? Yeah, we have... Literally, we do have a time for a question or two if anyone does want to ask anything. Uh, oh, fantastic.

[26:42](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1602s) I have a question on the slide.

[26:44](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1604s) On the slide.

[26:44](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1604s) Will they be available?

[26:45](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1605s) Oh. Sorry, you... Oh, you, you wanna get a copy of the slides?

[26:49](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1609s) Yes.

[26:49](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1609s) Okay. He wants a copy of the slides. Yeah. Yeah. Sure. Okay, cool.

[26:55](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1615s) Oh, sorry.

[26:56](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1616s) I have a question.

[26:56](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1616s) Yeah, go, go for it.

[26:57](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1617s) Okay. So you, you had mentioned guardrails as a service, and this is one of the interesting paradigms that we're looking into as well and trying to, uh, sort of figure out how to best serve guardrails as a service.

[27:08](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1628s) Did you see or is, is there a certain leaning towards you put guardrails in front of all models that you serve, and then, like in your model card, you say, "This is the model.

[27:19](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1639s) These are all the guardrails in front"? Or do you expect product teams to consume the guardrails but leave it in their remit to go and consume them properly?

[27:31](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1651s) Yeah. So, so what we've seen is that the central governance teams will put the generic rules in place. And then depending on the use case and whatever they're putting, the, each of the teams put their own rules on top of that.

[27:43](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1663s) So there's a kind of self-servicing, but also like a centralized component of rules. Not every team has to duplicate, uh, to go from there. Cool. Thank you very much.

[27:55](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1675s) All right. Now shall we get set up? [upbeat music]
