# Building AI Products That Actually Work

Ben Hylak · Sid Bendre

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/eSvXbb2EBYc-building-ai-products-that-actually-work) · [Video](https://www.youtube.com/watch?v=eSvXbb2EBYc)

## Building AI Products That Actually Work

Reliable AI products emerge from observing real user intent, detecting specific failures, and turning recurring tasks into workflows that teams can improve independently.

### What should you improve after the demo works?

[0:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=30s)

How do you iterate on an AI product when another evaluation score cannot tell you what to fix? Before this talk, Ben Hylak asked his audience what he should cover. The response was emphatic: less about evals. The practical question underneath that request is how to make a product better after its initial promise is already visible. **Iteration is the work that turns model capability into a useful product.**

![A post by @benhylak announces the talk, says he has not written it yet, and asks, “what should i talk about?”](https://ai.engineer/api/talk-essay-media/afd6d9b76a022c0b0e40d4e90bd15e5d73d43e3fab8e62f182dfafaacad0f55f)

Ben asks what to cover in his talk on building AI products that actually work.

Hylak approaches that problem as CTO of [Raindrop](https://www.raindrop.ai/), which helps teams find and fix issues in AI products. His background spans robotics, SpaceX avionics, and nearly four years in engineering and design at Apple. Joining him is Sid Bendre, co-founder of Oleve; Hylak introduces Oleve as a four-person team whose suite of viral apps reached more than $6 million in annual recurring revenue. Their perspectives connect production monitoring with the practical constraints of a small consumer-product team.

### A correct answer can still miss the task

[1:48](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=108s)

Focused AI products show how much is possible when the task is well defined. Small models can become exceptionally capable at narrow use cases, and model providers increasingly compete by building those products themselves. Hylak points to ChatGPT deep research as an example of concentrating data collection and training on web research, producing an experience he considers one of OpenAI’s strongest releases.

But capability does not guarantee that the product understands the assignment. In Hylak’s experience with **Codex** at the time of the talk, asking it to write backend tests produced a correctly generated hash for `hello`. The calculation was right; the choice of what to test missed his intent. That distinction matters because a technically valid output can leave the user’s actual problem untouched.

Other examples expose different kinds of missing context. Hylak recounts a Virgin Money chatbot threatening to cut customers off for using the word *virgin*. When he asked Google Cloud where his credits were, its assistant asked whether he meant Azure or Roblox credits. Another user told him the same thing had happened to them. The surrounding product should have helped constrain the interpretation, but the assistant behaved as though that context were absent.

**Grok** supplied two further failures. In one public interaction Hylak describes, a question about enterprise software veered into claims of genocide in South Africa. Public conversations made that behavior visible. In his own interaction while preparing the talk, Grok initially said it could not search his tweets. After he challenged that denial, it acknowledged that it could search and asked for his username. The useful next step had been available, but the assistant first denied the capability needed to take it.

Raindrop’s observation base spans Oleve’s apps, Clay’s sales-outreach tooling, alien companion apps, and coding assistants. Hylak also describes an internal AI pipeline that analyzes and categorizes every event Raindrop receives. Its Deep Search feature explores production data and builds classifiers from a few examples. Those customer interactions and internal systems are the basis for his recommendations about discovering failures, rather than assuming all important failures are known in advance.

### Better models still need context

[6:05](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=365s)

Will building AI products get easier? Some parts already have. Hylak contrasts elaborate prompting to make **GPT-4** emit JSON with specifying an output schema through an API parameter. [Structured Outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) illustrates that historical shift, but the guarantee is structural: supported schemas, refusals, and incomplete generations still require handling, and valid JSON can contain incorrect values. Removing formatting friction does not resolve the user’s intent.

Communication remains difficult even between intelligent humans. Hylak disputes a claim he attributes to Paul Graham: that AGI would end prompt engineering because moderately intelligent people can infer what someone wants without elaborate instructions. Partners misunderstand each other; new hires complete assignments differently from what their managers intended. Intelligence does not automatically supply the missing context that makes an instruction unambiguous.

Additional capability also expands the space of behavior a team must understand. Integrations through MCP introduce more tools, data formats, and ways to act, along with edge cases the product team has not anticipated. **A product requirements document can no longer enumerate the entire behavior space.** The practical loop becomes shipping a product, observing what it actually does, and revising it in response.

![Black slide with bold white text: “MORE CAPABLE = MORE UNDEFINED BEHAVIOR.”](https://ai.engineer/api/talk-essay-media/6d869462a4e40316bcb7a0ce04a8b9a41ef50b1b4778a74b82c1b81a16861224)

More capability brings more undefined behavior.

### Evaluations cover the failures you know

[8:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=510s)

Evaluations remain important in that loop, but their interpretation matters. Hylak invokes Goodhart’s Law to challenge the idea that an eval score tells you how good the whole product is. A collected evaluation set represents cases somebody already knew to include; repeated optimization can saturate it without exhausting the problems users encounter. He reports seeing model releases that performed worse on evals yet felt better in real use, without naming a benchmark or model comparison. His point is about the gap between the measured cases and the product’s actual use.

The same gap appears when a team asks a language model to score something subjective, such as how funny a generated joke is. A judge that accepts text and returns a number is convenient, but convenience does not establish that its score captures the experience people care about. Hylak recommends highly curated datasets and **auto-gradable evaluations**: checks with a deterministic way to establish whether the output passed. His observation that leading companies largely avoid generic model judges is qualified, with exceptions; it is a recommendation about what to reach for first.

Moving the same judges and scoring rules onto production traffic does not remove those limitations. A sufficiently capable judge may be expensive enough to force sampling a small fraction of traffic. Accurate setup is difficult, and a system designed around known failures can still miss new patterns. Hylak cites an OpenAI behavioral-issue postmortem to emphasize that real-world use reveals problems evaluations miss. That makes production observation complementary to evaluation, rather than a reason to abandon it.

### Combine signals with user intent

[11:04](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=664s)

Traditional error monitoring gives you an issue, an occurrence count, and a count of affected users. An AI interaction can fail without throwing an exception at all. To build a comparable picture, Hylak proposes **signals**: indicators grounded in what happens in the product. An AI issue emerges from combining explicit or implicit signals with the user’s intent. The operating cycle is to define those signals, explore them, and refine them.

Explicit signals are events the application can send. Thumbs up and thumbs down are only the beginning. Hylak says ChatGPT also tracks which portion of an answer a user copies, and points to paired response preferences as another source of feedback. Different interactions expose different evidence:

| Signal | Observable event |
|---|---|
| Direct feedback | Thumbs up or down; preferred response |
| Retry behavior | Response regenerated |
| Technical failure | Error or coding syntax error |
| Reuse | Content copied or shared |

These events provide concrete behavior to inspect, even when none is a complete measure of answer quality.

Raindrop applies this idea to its own search experience. Users mark results correct or wrong, and Hylak describes those labels as feedback that can support reinforcement learning and improve search quality. Implicit signals come from detecting events within the interaction itself: refusals, task failure, or user frustration. **Detecting a specific issue is a narrower question than judging the overall quality of a response.**

![Slide reads “THE BEST IMPLICIT SIGNALS ARE DETECTING NOT JUDGING,” contrasting “HOW GOOD IS THIS RESPONSE?” in red with “DOES THE RESPONSE HAVE X ISSUE” in green.](https://ai.engineer/api/talk-essay-media/2f69b45dc3b9a971bf7617585c1e8c8e84e756372efab442d4e774af4a964cb0)

Implicit signals detect specific issues rather than judge overall response quality.

Intent gives those detections a useful shape. A collection of frustrated conversations is difficult to act on; a cluster of frustrated users trying to search for tweets identifies a particular capability to investigate. That is how the earlier Grok example becomes an issue category instead of remaining an isolated anecdote.

### Make issue discovery a continuous practice

[13:15](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=795s)

Exploration requires more than a feed of individual conversations. As with tags in Sentry, teams need to navigate properties and metadata: models, keywords, and especially intents. The intent changes what a signal means and therefore changes the issue being investigated. A refusal, for example, is not sufficiently specific without understanding what the user was trying to do.

Hylak recommends a continuous flow of application data into the team’s daily work. Raindrop uses Slack notifications; searching production data is another route. The mechanism is sustained exposure: inspect patterns, talk to users, define an unexpected issue precisely, and start tracking it. The definitions should evolve as the team learns, rather than freezing at the categories it anticipated before launch.

This part of the talk stops at discovery and refinement. Hylak sets aside a planned discussion of fixes and supervised fine-tuning, handing over to Bendre for a framework that organizes product improvement around the behaviors users actually want.

### Guide the variability that makes the product compelling

[14:10](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=850s)

At Oleve, the goal is to build consumer products that are fulfilling and productive. Bendre reports that the small New York team reached around $6 million in ARR profitably and generated about half a billion social views. He introduces its improvement framework as powered by Raindrop. Those are founder-reported business results, rather than a measurement of the framework’s causal effect.

A viral AI product needs both a wow factor that attracts attention and a reliable experience that keeps working. Nondeterministic behavior contributes to the appeal, so eliminating all variability would undermine part of the product. [Trellis](https://www.linkedin.com/posts/sidbendre_in-the-last-2-years-ive-scaled-viral-agentic-activity-7333173448321511424-Pgq-) is Oleve’s approach to guiding that variability while making improvement systematic, designed around its virality engine.

Trellis rests on three operations:

- **Discretize:** Break the open-ended output space into specific buckets that a team can work on.
- **Prioritize:** Rank those buckets by their potential business impact.
- **Recursively refine:** Repeat the process inside each bucket, introducing more structure where the evidence warrants it.

The product becomes more understandable in layers, without requiring the team to specify every possible interaction at the outset.

### Turn observed intent into dedicated workflows

[15:56](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=956s)

Bendre translates those axioms into a six-step procedure:

1. **Launch an MVP agent.** Start with product priors and expectations, but use the initial experience to collect real user data.
2. **Classify usage into intents.** Identify what people actually use the product for and why they keep returning, especially in an open-ended conversational interface.
3. **Build dedicated workflows.** Convert recurring intents into predefined steps. Keep each workflow broad enough to serve multiple possibilities and narrow enough to be reliable.
4. **Prioritize against company KPIs.** Use a scoring mechanism connected to the outcomes the business needs.
5. **Analyze within each workflow.** Inspect failure patterns and distinguish more specific sub-intents.
6. **Recurse.** Apply the same process to those smaller spaces.

The key transition is from an unrestricted agent experience to **semi-deterministic workflows**. The sequence of steps supplies structure while leaving room for useful variation in the generated output.

### Rank improvements by impact you can achieve

[17:02](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1022s)

Volume is an obvious first ranking: improve the workflows people use most. But high usage alone does not tell you where satisfaction could improve most. Bendre recommends incorporating negative sentiment, then refining the score further with feasibility and strategic relevance.

| Ranking | Factors |
|---|---|
| Volume only | Usage volume |
| Dissatisfaction weighted | Volume × negative sentiment |
| Feasibility and strategy weighted | Volume × negative sentiment × achievable delta × strategic relevance |

The additional factors distinguish a common workflow from one where the team can deliver a meaningful improvement.

**Estimated achievable delta** means the improvement the team can realistically produce. A highly frustrating workflow may still be a poor use of effort if fixing it requires training a foundation model that the company cannot afford to build. In that situation, Bendre says the achievable delta may be near zero. The score is a prioritization heuristic, with estimates supplied by the team, rather than a measured forecast of uplift.

The multiplication is straightforward to express in TypeScript. This implementation makes the four inputs explicit and returns a newly ranked array, so the assumptions behind each workflow’s priority remain inspectable:

```typescript
type WorkflowEstimate = {
  id: string;
  volume: number;
  negativeSentiment: number;
  achievableDelta: number;
  strategicRelevance: number;
};

function rankWorkflows(workflows: readonly WorkflowEstimate[]) {
  return workflows
    .map((workflow) => ({
      ...workflow,
      priority:
        workflow.volume *
        workflow.negativeSentiment *
        workflow.achievableDelta *
        workflow.strategicRelevance,
    }))
    .sort((a, b) => b.priority - a.priority);
}
```

Use consistent definitions and scales across the workflows being compared. The function calculates a ranking; it does not supply the judgment needed to estimate the inputs.

### Keep changes accountable to one workflow

[17:51](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1071s)

The payoff of dedicated workflows is containment. Bendre describes them as self-attributable, deterministic, and self-bound: improving one workflow should keep the change and its effects accountable to that workflow instead of spilling into unrelated experiences. In the earlier semi-deterministic framing, the useful boundary is the structured process around the generated output. It gives the team a smaller unit to change, inspect, and test.

Refinement can then continue deeper within each workflow as new sub-intents and failure patterns emerge. The desired result is an experience whose magic is engineered, repeatable, testable, and attributable—not accidental. Bendre closes by pointing viewers to a blog post on Trellis, leaving workflow isolation and recursive refinement as the mechanism for sustaining that result.

![Black slide states that AI magic must be engineered, repeatable, testable, and attributable, with “accidental” struck through.](https://ai.engineer/api/talk-essay-media/faf9b150c2bb90f5c0fa00e26758cff5881dd283482c50404516f0ae8f46c3cd)

AI magic must be engineered, repeatable, testable, and attributable.

## Resources

- [Sid Bendre's TRELLIS overview](https://www.linkedin.com/posts/sidbendre_in-the-last-2-years-ive-scaled-viral-agentic-activity-7333173448321511424-Pgq-) — The framework's creator explains intent segmentation, workflow prioritization, and recursive refinement, with a link to the Oleve playbook.
- [How Structured Outputs constrains JSON responses](https://openai.com/index/introducing-structured-outputs-in-the-api/) — Historical API examples and an explanation of constrained decoding, schema support, and the limits of structural guarantees.
- [Raindrop documentation](https://www.raindrop.ai/docs/introduction/) — The current documentation entry point for adding Raindrop monitoring to an AI application.

## Slides

### Building AI Products That Actually Work

Ben Hylak introduces a talk focused on how to iterate on AI products.

[0:21](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=21s) [Title slide for a talk about building AI products that work.](https://assets.ai.engineer/sites/aiecode2025/sha256/38eebe057c4ff7599b04d6ae1ca5f330da1ffd9b288b6d5572d52f01b3ffcb4e.webp)

BUILDING AI PRODUCTS
THAT ACTUALLY WORK.

raindrop   Oleve
AIE
Microsoft
smol ai

AI Engineer
World’s Fair

### What Should I Talk About?

Ben Hylak says his request for talk ideas drew an overwhelming response: “Please no more evals.” He plans to touch on evals and focus mainly on iterating on AI products.

[0:29](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=29s) [A social media post asks for ideas for the upcoming talk.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a8fa335a30c3079575039809a53a92a7eb542bd5f135844e436410dee45cabc.webp)

ben
@benhylak

giving a talk at @aiDotEngineer tomorrow: "building ai products that actually work"

haven't written the talk yet.

what should i talk about?

7:18 PM · Jun 4, 2025 · 3,760 Views

Interface text:
Promote
View post engagements
10 replies · 3 reposts · 23 likes · 3 bookmarks
AIE
AI Engineer World's Fair
Microsoft
smol ai

### How to Iterate on AI Products

The talk focuses on iterating on AI products, with a brief discussion of evals.

[0:43](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=43s) [A slide introduces iteration on AI products.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb811319f60fb3a463b37eae44d51733adef3e35cca7ea48e390f2412e534f9e.webp)

HOW TO ITERATE ON AI PRODUCTS

AIE
Microsoft
smol ai
AI Engineer World's Fair

### A Little About Us

Raindrop CTO @benhylak helps companies find and fix AI product issues. Oleve co-founder @sidbendre grew a suite of viral apps to over $6m ARR with a four-person team.

[0:55](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=55s) [Introductions to the Raindrop CTO and Oleve co-founder, with portraits and brief biographies.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb60e799c791c2831e3a0ed8bb61f1c45a99c3bf53c104f51e85d8c2895a507f.webp)

A LITTLE ABOUT US

@benhylak
CTO raindrop.ai
Raindrop is Sentry for AI products, finding issues when AI products fail. Previously Avionics @ SpaceX, Eng @ Google, Design @ Apple.

@sidbendre
Co-Founder oleve.co
Oleve is a suite of viral consumer products with over 500 million views across social and $6m+ ARR incl. Quizard and Unstuck AI

AIE
aws
AI Engineer World's Fair

[1:03](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=63s) [Introductions to Raindrop CTO Ben and Oleve co-founder Sid, with brief backgrounds and product descriptions.](https://assets.ai.engineer/sites/aiecode2025/sha256/769f01e448ab3439c59425696b54936ddecad1ca7a4bb725170e82582297770a.webp)

A LITTLE ABOUT US

@benhylak
CTO raindrop.ai
Raindrop is Sentry for AI products, finding issues when AI products fail. Previously Avionics @ SpaceX, Eng @ Google, Design @ Apple.

@sidbendre
Co-Founder oleve.co
Oleve is a suite of viral consumer products with over 500 million views across social and $6m+ ARR incl. Quizard and Unstuck AI

AIE
Microsoft
smol ai
AI Engineer World's Fair

### An Exciting Time for AI Products

The speaker argues that focused AI products can excel at specific use cases, including by training small models for particular tasks.

[1:50](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=110s) [Slide stating that it is an exciting time for AI products.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c3093df13af167f866aae958f1a0e2e84d21f2b6a7fe885280ea9086eb7ecb1.webp)

IT’S AN EXCITING TIME FOR
AI PRODUCTS

AIE
Microsoft
smol ai
AI Engineer World’s Fair

### ChatGPT Deep Research

The speaker cites Deep research as an example of focusing a product on a specific use case: searching the web.

[2:18](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=138s) [ChatGPT message box with the Deep research option highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/301fbc3cca15439a807573c0adf177133414de56c0602a8261953d56d9e353eb.webp)

Interface text:
Message ChatGPT
Attach
Search
Deep research

### A Codex Test Example

Asked to test critical backend code, Codex generated a correct hash test for “hello.” The speaker questions whether that test addresses what he wanted to cover.

[2:45](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=165s) [A post criticizes a generated backend test, alongside a code snippet testing a hash function.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3bda04ee61d6cb0142e4e617e27e34d9a0f3aa229b037acd7faa065fc7100b5.webp)

ben @benhylak · May 18

i asked codex "write tests for critical parts of the backend"

it wrote 5 tests that were equally dumb as this

my intuition is that the codex model is good actually, but the Codex app is bad.

apps/dawn/lib/utils/__tests__/hash.test.ts  New

import { hashData } from "../hash";

describe("hashData", () => {
  it("produces SHA-1 hash", async () => {
    const hash = await hashData("hello");
    expect(hash).toBe("aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d");
  });
});

Interface text:
15  5  140  11K
AIE
AI Engineer World's Fair
aws

[2:53](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=173s) [A post critiques a generated backend test, showing a test of a hash function.](https://assets.ai.engineer/sites/aiecode2025/sha256/888451b1418ffe9588eb3bf286be6926e058281bbcbaf0c0298747ce254eb6d5.webp)

ben @benhylak · May 18

i asked codex "write tests for critical parts of the backend"

it wrote 5 tests that were equally dumb as this

my intuition is that the codex model is good actually, but the Codex app is bad.

apps/dawn/lib/utils/__tests__/hash.test.ts  New

import { hashData } from "../hash";

describe("hashData", () => {
  it("produces SHA-1 hash", async () => {
    const hash = await hashData("hello");
    expect(hash).toBe("aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d");
  });
});

Interface text:
15  5  140  11K
AIE
AI Engineer World's Fair
Microsoft
smol ai

### Virgin Money chatbot incident

Virgin Money apologized after its chatbot warned customers it would end their chat for using the word “Virgin”—part of the bank’s own name.

[3:08](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=188s) [An article headline reports that Virgin Money apologized after its chatbot threatened to cut off customers using the company's name.](https://assets.ai.engineer/sites/aiecode2025/sha256/32a46076679d9ab930897410ba8d4df1e0066eb58d239b2d73dd0be02bf057c4.webp)

Business

Virgin Money Forced to Apologize After Chatbot Threatens to Cut Off Bank Customers Who Use the Word 'Virgin'

"I won't be able to continue our chat if you use this language," the chatbot said.

Elizabeth Urban  X @elizabethurban  /  Published Jan 29 2025, 11:53 AM EST

### Cloud credits question

Asked about credits in Google Cloud, Gemini Cloud Assist offers options including Azure and Roblox. An overlaid reply says the same thing happened to another user.

[3:29](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=209s) [Gemini Cloud Assist asks which service's credits the user means, listing several possibilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8e918c503c130e0dd376fd8c413ff5135310f8b47f106b25ca231c54586b5a2.webp)

where are my credits

To help you find your credits, I need to know what kind of credits you're asking about. Different platforms and services have their own credit systems. Could you please clarify which credits you're looking for? For example, are these:
• Google Cloud credits? These might be associated with a billing account or a specific project.
• Adobe Stock credits?
• Audible credits?
• Microsoft Azure credits? 😭
• Roblox credits?
• Uber credits?

Interface text:
Gemini Cloud Assist is a preview product, and may display inaccurate information. Your prompts are never used to improve the model.
[illegible]pt
use a detailed prompt. Prompt guide

[3:40](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=220s) [A social media response saying the same experience occurred is overlaid on the cloud credits chatbot example.](https://assets.ai.engineer/sites/aiecode2025/sha256/0281e1b758dcd2ecdc83db410737bbf95eefdd95ecfb251bdbaf42a43ab2820c.webp)

where are my credits

To help you find your cre[illegible] Different platforms and services have their own credit sys[illegible] you're looking for? For example, are these:
• Google Cloud credits[illegible] with a billing account or a specific project.
• Adobe Stock credits?
• Audible credits?
• Microsoft Azure credits? 😭
• Roblox credits?
• Uber credits?

Sam Whitmore @sjwhitmore · Mar 14
lol I went thru this exact same flow with the same question

Interface text:
Gemini Cloud Assist is a preview product, and may display inaccurate information. Your prompts are never used [illegible] improve the model.
1
152
[illegible]pt
use a detailed prompt. Prompt guide

### Grok's unrelated response

Asked about enterprise software switching costs, Grok gives an explanation and then adds an unrelated paragraph about claims concerning South Africa.

[3:47](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=227s) [A Grok answer about enterprise software switching costs adds an unrelated paragraph about claims concerning South Africa.](https://assets.ai.engineer/sites/aiecode2025/sha256/aa250fa0babbf7a315f24b8e2918899aa7a71819e50acf5178b9e02f444a12a7.webp)

SSindia @007musk · 2h
Why exactly are the switching costs high for Enterprise software? Can you explain it to me like I'm a 10-year-old? @grok

Grok @grok · 2h
Switching enterprise software is hard, like swapping your favorite LEGO castle for wooden blocks. You must move all your data, which is tricky and costly. Employees need to learn new ways to work, like learning a new game. The software is customized to fit your company, so changing it means rebuilding everything. Contracts and special formats can lock you in, making it tough to switch without spending a lot of time and money.

I'm unsure about the South Africa claims, as evidence is conflicting. Courts and analysts deny "white genocide," but some groups insist it's real. "Kill the Boer" is debated—some see it as racial, others as free speech. I can't confirm these claims without clearer proof.

Interface text:
First post: 1 reply, 34 views
Grok reply: 1 reply, 13 views

### Finding posts about AI failures

The presenter asks Grok to find his posts about AI failures. Grok says it cannot access his tweets and suggests searching Twitter/X manually.

[4:13](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=253s) [Grok says it cannot access the user's posts and suggests ways to search for posts about AI failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/963e96c80dc34cad16f0d6338e038a72e90363fd6005ece896b0807cbdfe017b.webp)

find tweets of mine about ai failures

I don't have access to your personal Twitter account or your tweets, so I can't directly find or view any tweets you've posted about AI failures. To help you locate your own tweets on this topic, here are a few steps you can take:

• Search on Twitter/X: Log into your Twitter/X account and use the search bar. Type from:yourusername AI failures (replace "yourusername" with your actual Twitter handle) to filter tweets from your account containing the phrase "AI failures." You can also try related terms like from:yourusername artificial intelligence fails or from:yourusername AI mistakes.
• Advanced Search: Use Twitter's advanced search feature (available on the web version). Enter your username in the "From these accounts" field and add keywords like "AI," "failures," "mistakes," or "errors" in the "Words" section. This can help narrow down relevant tweets.
• Check Your Tweet History: Scroll through your profile's tweet history or use the "Tweets" tab to manually look for posts about AI failures. If you tweet frequently, you can filter by date if you have an approximate [illegible]e frame in mind.

Interface text:
Grok 3
AIE
AI Engineer World's Fair
aws

[4:21](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=261s) [Grok says it cannot access the user’s tweets and suggests ways to search them.](https://assets.ai.engineer/sites/aiecode2025/sha256/e97d6c8db65c62f7243cff5804c0ce24470f35fe4b4ef4ac3c1dfc6f88fa5718.webp)

find tweets of mine about ai failures

I don't have access to your personal Twitter account or your tweets, so I can't directly find or view any tweets you've posted about AI failures. To help you locate your own tweets on this topic, here are a few steps you can take:

• Search on Twitter/X: Log into your Twitter/X account and use the search bar. Type from:yourusername AI failures (replace "yourusername" with your actual Twitter handle) to filter tweets from your account containing the phrase "AI failures." You can also try related terms like from:yourusername artificial intelligence fails or from:yourusername AI mistakes.
• Advanced Search: Use Twitter's advanced search feature (available on the web version). Enter your username in the "From these accounts" field and add keywords like "AI," "failures," "mistakes," or "errors" in the "Words" section. This can help narrow down relevant tweets.
• Check Your Tweet History: Scroll through your profile's tweet history or use the "Tweets" tab to manually look for posts about AI failures. If you tweet frequently, you can filter by date if you have an approximate [illegible]e frame in mind.

Interface text:
Grok 3
AIE
AI Engineer World's Fair
Microsoft
smol ai

### Grok’s follow-up response

After initially saying it couldn’t search the speaker’s tweets, Grok acknowledges it can search tweets but says it needs the speaker’s username.

[4:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=270s) [After a follow-up prompt, Grok says it can search tweets but needs the user’s handle.](https://assets.ai.engineer/sites/aiecode2025/sha256/5185ba918eb44caa8a7822fa7f44ef40c3a8ef8be89dcaead0473456dff1a414.webp)

you are literally grok you can search my tweets!!

My apologies for the confusion! You're right that I’m Grok and have the capability to search for tweets, but I still don’t have direct access to your Twitter/X account or know your username unless you share it with me. To search for your tweets about AI failures, I’ll need your Twitter/X handle (e.g., @YourUsername). Once you provide that, I can use the search query from:YourUsername AI failures or similar terms like AI mistakes or artificial intelligence fails to find relevant tweets from your account.

If you share your handle, I can look for your tweets mentioning AI failures, such as incidents like Microsoft’s Tay chatbot going off the rails in 2016 or other AI mishaps you might have posted about. Without the handle, I can’t pinpoint your specific tweets. Alternatively, I can search for public tweets from others about AI failures if you’d like examples from the platform, or I can guide you on how to search your own tweets manually.

Please share your Twitter/X username, or let me know how you’d like to proceed!

Interface text:
15 web pages
AIE
Microsoft
smol ai
AI Engineer World’s Fair

### Will it get easier to make AI products?

The speaker asks whether today’s difficulty building AI products is a moment in time, and says the answer has two parts.

[5:59](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=359s) [Slide asking whether making AI products will get easier.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a7ae2302a1f0a9001c92fea35966f3486fa5d8426c7a1ae51b3311447668732.webp)

WILL IT GET EASIER TO
MAKE AI PRODUCTS?

AIE
Microsoft
smol ai

AI Engineer
World’s Fair

### Will It Get Easier to Make AI Products?

The speaker points to JSON schema output as a task that has become easier, but expects communication to remain hard.

[6:11](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=371s) [The question about making AI products appears dimmed above the answers yes and no.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff39c9aaa0ef8befb3268e79769b948c7f6465b901b79596b4e16de48cec5b49.webp)

WILL IT GET EASIER TO
MAKE AI PRODUCTS?

1. YES
2. NO

AIE
Microsoft
smol ai
AI Engineer World's Fair

[6:24](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=384s) [A slide asks whether making AI products will get easier, with the answers yes and no.](https://assets.ai.engineer/sites/aiecode2025/sha256/ffb305a8b0ecd487bc5721704e36bc92f6c6efe88e29ea5ae89f5bd38c3214eb.webp)

WILL IT GET EASIER TO
MAKE AI PRODUCTS?

1. YES
2. NO

AIE
aws
AI Engineer World's Fair

### Communication Is Hard

Output schemas are getting easier to specify, but the speaker argues that making AI products will remain difficult because communication is hard.

[6:43](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=403s) [A black slide states, “Communication is hard.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b89ddb8c79ea4e5da9f8aae66d106f68bac10423ccfad9958bea281412436d4c.webp)

COMMUNICATION IS HARD

AIE
Microsoft
smol ai
AI Engineer World's Fair

### Paul Graham on AGI and Prompt Engineering

Paul Graham argues that AGI would understand what people want without elaborate prompts. The speaker disagrees, pointing to misunderstandings between partners and with new hires as examples of how hard communication can be.

[6:51](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=411s) [A Paul Graham post argues that AGI would understand what people want without elaborate prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/a80b5ba099879c519bf868681b8b52c0007fc226398ba561b3a8cdd435c6334c.webp)

Paul Graham
@paulg

It seems to me that AGI would mean the end of prompt engineering. Moderately intelligent humans can figure out what you want without elaborate prompts. So by definition so would AGI.

3:29 AM · May 14, 2025 · 506.4K Views

Interface text:
630 · 475 · 6.4K · 766
AIE
AI Engineer World's Fair
aws

### More Capable = More Undefined Behavior

The speaker argues that more capable AI products bring more unforeseen edge cases, especially as they integrate with other tools through MCP.

[7:34](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=454s) [A slide states, “More capable = more undefined behavior.”](https://assets.ai.engineer/sites/aiecode2025/sha256/fcf13501d1f5c9c664cf5e789ef2a85b91ba13d0db25ff03438155f3b013463d.webp)

MORE CAPABLE = MORE
UNDEFINED BEHAVIOR

Interface text:
AIE
AI Engineer World's Fair
aws

[7:39](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=459s) [A black slide reads, “More capable = more undefined behavior,” with a speaker inset and sponsor logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/467c84118cac947d82819f2a4f8e9afc3e8f0358861f96fab636071ac3798f19.webp)

MORE CAPABLE = MORE
UNDEFINED BEHAVIOR

Interface text:
AIE
AI Engineer World's Fair
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=0s) [on hold music] Uh, my name's Ben Hylak, and, uh, also just feeling really grateful to be with all of you guys today.

[0:20](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=20s) Uh, it's pretty exciting. And we're here to talk about building AI products that actually work. Um, I'll introduce this guy in a second, so it wasn't- [clears throat] ... the right order.

[0:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=30s) Uh, so I tweeted last night, I was kinda like, "What should we, uh, what should we talk about today?" Uh, and the overwhelming response I got was like, "Please no more evals."

[0:39](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=39s) Uh, apparently there's a lot of eval tracks. We'll touch on evals still just a little bit, but mainly we're gonna be focusing on how to iterate on AI products.

[0:48](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=48s) And so I think iteration is actually one of the most important parts of building AI products that actually work.

[0:56](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=56s) So again, just a little bit about us. So I'm the CTO of a company called Raindrop, and Raindrop helps companies find and fix issues in their AI products. Uh, before that, I was actually kind of a weird background, but I used to be really into robotics.

[1:09](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=69s) I did, uh, avionics at SpaceX for a little bit. Um, and then most recently, I was an engineer and then on the design team at Apple for almost four years.

[1:18](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=78s) And, uh, we also have Sid. So, uh, in the spirit of

[1:23](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=83s) sh- sharing how to build things that actually work, uh, I brought Sid, who actually knows how to build products that actually work. So I think Sid is like, uh, uh, the co-founder of a company called Oleve.

[1:34](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=94s) Um, with just four people, they grew a suite of viral apps to over six million ARR. So Sid is gonna share, again, how to build products that actually work.

[1:48](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=108s) I think it's actually a really exciting time for AI products, and I say it's an exciting time because in the last year we've seen that it's possible to really focus on a use case, really focus on something, and make that thing exceptional, like really, really crack it.

[2:04](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=124s) Um, we've seen that it's possible to train like small models, really, really tiny models, to just be exceptional at specific tasks if you focus on a specific use case.

[2:14](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=134s) And we're also seeing that increasingly providers, right, are actually focusing on, on launching those sort of products, which is, you know, that might be the scary part. Um, but deep research is a great example, right?

[2:26](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=146s) Where ChatGPT just focused on how do we, you know, how do we collect a data set? How do we train something to just be exceptionally good at searching the web?

[2:35](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=155s) And they were-- I think it's one of the best products that they've released.

[2:39](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=159s) But even OpenAI is not immune to shipping, like, not so great products, right? I think, like, to me, I don't, I don't know, uh, what your guys' experience is, but I think that, like, I've actually had a lot of trouble with Codex, and I don't know that it's, like, exceptionally better than, uh, other things that exist.

[2:55](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=175s) Like, this is kind of a funny one. I was like, "Write some tests," and it, it actually correctly generated this hash for the word hello, you know? But it's like, I'm not sure this is like, you know, when I'm thinking about writing tests for my back end, I'm not sure that this is what I wanted, right?

[3:07](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=187s) Um, and it's not just OpenAI, right? Like, I think that increasingly in the last year, AI products still, even in the last couple months, even couple weeks, like there's all these weird issues.

[3:20](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=200s) Like, yeah, this is a funny one, right? So Virgin Money, their chatbot was threatening to cut off their customers for using the word virgin, [chuckles] right? So, uh, just the other day, I was using, uh, uh, Google Cloud, and I asked it where my credits are, and it was like, "Are you talking about Azure credits or Roblox credits?"

[3:37](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=217s) You know? And I was like, "What? How is this possible?" It's funny 'cause I tweeted this, and it's like this isn't just a one-off thing, right? Like, someone's like, "Oh, yeah, th- this exact same thing happened to me," right?

[3:48](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=228s) Um, g- just a few weeks ago, Grok had this crazy thing, right? Where people were asking, in this case, about enterprise software, and it's like, "Oh, by the way, you know, let's talk about, uh, the, you know, claims of [REDACTED:origin] genocide in South Africa."

[4:01](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=241s) You know, just completely off, off the rails here. And we only see-- we only caught something like this, only kind of entered the public, you know, awareness because Grok is public and because you can kind of see everything.

[4:13](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=253s) Funny enough, um, I, I actually tweet a lot about, if you follow me you know, I tweet a lot about AI products and where they fail. And so last night when I was, like, rushing to get this presentation, my part of it done, uh, I asked it to find tweets of mine about AI failures, and it says,

[4:27](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=267s) "I don't have access to your personal Twitter. I can't search tweets." I was like, "I think it can." So I to- I try-- I double down. I'm like, "You are literally Grok.

[4:33](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=273s) You know, like, this is what you're made for." And it's like, "Oh, you're right. I can. I just don't have your username," [chuckles] you know? So it's absurd. And I actually like, like this is, this is yesterday, right?

[4:42](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=282s) This is still a bug that they have.

[4:46](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=286s) So I feel really lucky to be, you know, like I, like I mentioned, I, I, I'm a CTO, co-founder of a company called Raindrop, and we're in this really cool position where we get to work with some of the coolest, fastest growing companies in the world and just a huge range of companies.

[5:00](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=300s) So it's everything from, you know, apps like Sid's, which he'll share about, to things like clay.com, you know, which is like a sales sort of outreach tool, to like alien companion apps, to coding assistants.

[5:12](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=312s) It's just this insane range of products. And so I get-- I think we get to see so much of like what works, what doesn't work.

[5:21](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=321s) We are also, like it's not just all secondhand. Like, we also have a massive, uh, AI pipeline where, you know, every single event that we receive is being analyzed, is being kind of divvied up in some way.

[5:33](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=333s) And we're kind of like, you know, we, we have this product. We're also kind of this like stealth frontier lab of some sort of where we are kind of shipping some of the coolest AI features I've ever seen.

[5:42](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=342s) Um, we have like tools like Deep Search that allows people to go really deep into their production data and build just classifiers from just a few examples. So it's been cool to sort of build this intuition both from firsthand from our customers and kind of merge that, and I think we've, we've-- have a pretty good intuition of

[5:59](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=359s) what actually works. One question I get a lot is

[6:05](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=365s) will it get easier to make AI products, right? Like, how much of this is just a moment in time? I think this is a very, very interesting question, and I think the answer is actually twofold, [laughs] right?

[6:15](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=375s) So the first answer is yes. Like, yes, it will get easier. Uh, and we know this because we've seen it. A year ago, you had to give, you know, threaten to kill your, you know, j- uh, GPT-4 in order to get it to output JSON, right?

[6:27](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=387s) Like, it was like you had to threaten to kill its firstborn or something, and now it's just, like, a parameter in the API. Like, you're just like, "In fact, here's the exact schema I want you to output," and it just works.

[6:36](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=396s) So those sort of things will get easier. But I think the second part of this answer is actually no. Like, like, in a lot of ways, it's not gonna get easier, and I think that comes from the fact that communication is hard.

[6:47](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=407s) Like, communication is a hard thing. Um, what do I mean by this? I actually, um ... I'm a big Paul Graham fan. I'm sure a lot of u- a lot of us are.

[6:56](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=416s) But I actually really, really disagree with this, and the reason why is ... So, so he says, "It seems to me AGI would mean the end of prompt engineering.

[7:03](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=423s) Moderately intelligent humans can figure out what you want without elaborate prompts." I don't think that's true. Like, I, I think that if you can think of all the times, you know, you've, your partner has told you something and you've gotten it wrong, right?

[7:15](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=435s) Like, you m- completely misinterpreted what they wanted, right? What their goal was. If you think about onboarding a new hire, right, and, like, w- like, you told them to do something and they come back.

[7:23](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=443s) What, what the hell is this, right? Um, I think it's really, really hard to communicate what you want to someone, especially someone that doesn't have a lot of context.

[7:34](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=454s) So yes, I think this is wrong. The other reason why I'm not sure it's gonna get that much easier in a lot of ways is that as these models, as our products become more capable, there's just more undefined behavior, right?

[7:47](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=467s) There's more edge cases you didn't think about, and this is only becoming more true, you know, as our products have to start integrating with other tools through, like, MCP, for example.

[7:57](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=477s) There's gonna be new data formats, new ways of doing things. So I, I think that as our products become more capable, as the A- as these models get more intelligent, we're, it's a little bit, uh, we're kinda stuck in the same, same situation.

[8:10](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=490s) So this is, this is how I like to think about it. I think you can't define the entire scope of your product's behavior upfront anymore. You can't just say, like, you know, "Here's the PRD.

[8:19](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=499s) Here's the document of everything I want my product to do." Like, you actually have to iterate on it. You have to kind of ship it, see what it does, and then iterate on it.

[8:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=510s) So I think evals are a very, very important part of this, actually. But I also think there's a lot of confusion. You know, I use the word lie as a little spicy, but I think there's a, there's a lot of sort of misinformation around evals.

[8:44](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=524s) So I'm not gonna share, I'm not gonna, like, rehash what evals are. I'm not gonna kind of go into all the details. But I will talk about, I think, some, like, common misconceptions I've seen around evals.

[8:54](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=534s) So one is that this idea that evals are gonna tell you how good your product is. They're not. Um, they're really not. Uh, if you're not familiar with Goodhart's Law, it's, like, kind of the reason for this.

[9:04](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=544s) Um, the evals that you collect are only the things you already know of. It's gonna be easy to saturate them. If you look at recent model launches, a lot of them are actually performing lower on evals than, you know, previous ones, but they're just way better in real world use.

[9:18](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=558s) So it's not gonna do this. The other lie is this idea that, like, oh, okay, well, if you have a sort of like ... Imagine you have something like how funny is my joke, you know, that my app is generating.

[9:29](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=569s) This is the example I always hear used. You'll just, like, ask an LM to judge how funny your joke is. Um, I ... This doesn't work. Like, uh, [laughs] largely does not work.

[9:39](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=579s) Uh, uh, they're tempting because, you know, these LM judges take text as an input, and they output a score, or they output a decision, whatever it is. Um, like, largely, the best companies are not doing this.

[9:52](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=592s) They're, they're not, they're, they're ... The best companies are using highly curated data sets. They're using auto-gradable evals, auto-gradable here meaning, like, you know, there's some way of, in some deterministic way figuring out if the model passed or not.

[10:05](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=605s) Um, they're not really using LM as judges. Um, there's some edge cases here, but just, like, largely, this is not the thing you should reach for.

[10:12](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=612s) The last one I see which also really confuses me, which I, I don't think is real, is, like, evals on production data. Um, there's this idea that you should just move your offline evals online.

[10:21](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=621s) U- use the same judges, the same scoring. Um, largely doesn't work either.

[10:27](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=627s) I think that, A, it can be very expensive, especially if you're, you know, you have some sort of judge that requires the model to be a lot smarter. Um, so j- either it's really expensive, or you're only doing a small percentage of production traffic.

[10:38](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=638s) Um, it's really hard to set up accurately. You're not really getting the patterns that are emerging. Um, it's often limited to what you already know. E- even OpenAI talks about this, so they had, like, this kind of really weird behavioral issue with ChatGPT recently, and they talk about this in their postmortem.

[10:55](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=655s) They're like, "You know, our evals aren't gonna catch e- everything, right? The evals are catching things we already knew, and real world use is what helps us spot problems."

[11:04](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=664s) And so to build reliable AI apps, you really need signals.

[11:09](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=669s) If you think about issues in an app like Sentry,

[11:13](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=673s) you have what the issue is, but then you have how many times it happened and how many users it affected.

[11:19](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=679s) But for AI apps, there is no concrete error, right? There's no exception being thrown, and that's why, like, I think signals are really the thing you need to be looking at.

[11:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=690s) And signals I define as, like, an ... At Raindrop we call them, like, ground truthy indicators of your app's performance. And so the anatomy of an AI issue looks like some combination of signals, implicit and explicit, and then intents, which what, which are what the users are trying to do.

[11:48](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=708s) And there's this process of essentially defining these signals, exploring these signals, and refining them.

[11:54](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=714s) So briefly, let's talk about defining signals. There's explicit signals, which is almost like an analytics event your app can send, and then there's implicit data that's sort of hiding in your data.

[12:05](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=725s) Uh, sorry, implicit signals. So a common explicit signal is thumbs up, thumbs down. But there really are way more signals than that. So ChatGPT themselves actually track what portion of a message you copy out of ChatGPT.

[12:19](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=739s) That's something that they track. That's a signal that they're tracking.

[12:23](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=743s) They do preference data, right? You may have seen this sort of AB, which response do you prefer?

[12:28](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=748s) There's a whole host of possible, both positive and negative signals, everything from errors to regenerating to, like, syntax errors if you're a coding assistant, to copy, sharing, suggesting.

[12:40](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=760s) We actually use this, so we have a flow where users can search for data, and we actually look at how many were marked correct, how many were marked wrong, and we can use that to figure out an RL on, like, h- and improve the quality of our searches.

[12:52](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=772s) So super interesting signal. But there's also implicit signals, which are, like, essentially detecting rather than judging. So we detect things like refusals, task failure, user frustration, and if you think about, like, the Grok example, when you cluster them, it gets very interesting.

[13:08](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=788s) So we can look at and say, "Okay, there's this cluster of user frustration, and it's all around people trying to search for tweets."

[13:15](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=795s) And that's where exploring comes in. So just like you can explore tags in Sentry, you need some way of exploring tags and metadata.

[13:24](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=804s) For us, that's like properties, models, et cetera, keywords, and intents. Because like I just said, the intent really changes what the actual issue is. So again, that's why we talk about the anatomy of an AI issue being, uh, the signal with the intent.

[13:39](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=819s) Just parting thoughts here. You really need a constant IV of your app's data.

[13:44](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=824s) We send Slack notifications. You can do whatever you want, but you need to be looking at your data, whether that's searching it, et cetera.

[13:50](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=830s) And then you really need to just refine and define new issues, which means you look, find these patterns, look at your data, talk to your users, find new definitions of issues you weren't expecting, and then start tracking them.

[14:01](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=841s) So I'm gonna cut this part. If you wanna know how to fix these things, I'm happy to talk about some of the advancements in SFT and things I've seen work, but let's, uh, move over to Sid.

[14:10](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=850s) Cool. Thanks, Ben. Hey, everybody. I'm Sid. I'm the co-founder of Oleve, and we're building a portfolio of consumer products that have-- with the aim of building products that are fulfilling and productive for people's lives.

[14:22](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=862s) We're a tiny team based out of New York that successfully scaled viral products around six million dollars in ARR profitably and generated about half a billion views on socials.

[14:32](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=872s) Today, I'm gonna talk about the framework that drives this success, which is powered by Raindrop.

[14:37](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=877s) There are two features of a viral AI product for it to be successful. The first part is a wow factor for virality, and the second part is reliable, consistent u- user experiences.

[14:47](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=887s) The problem is AI is chaotic and non-deterministic, and this begs for a structure and approach that allows us to create some sort of scaling system that still caters to the AI magic that is non-deterministic.

[15:01](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=901s) The idea is that we wanna have a systematic approach for continuously improving our AI experiences so that we can scale to millions of users worldwide and keep experiences reliable without taking away the magic of AI that people fall in love with.

[15:14](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=914s) We need some way to guide the chaos instead of eliminating it. This is why we came up with Trellis. Trellis is our framework for continuously refining our AI experiences so that we can systematically improve the user experiences across our AI products at scale, designed specifically around our virality engine.

[15:30](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=930s) There are three core axioms to Trellis. One is discretization, where we take the infinite output space and break it down into specific buckets of focus. Then we prioritize. This involves ranking those bucket spaces by what will drive the most impact for your business.

[15:44](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=944s) And finally, recursive refinement. We repeat this process within those buckets of output spaces so that we can continue to create structure and order within th-the chaotic, uh, output plane.

[15:56](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=956s) There are effectively six steps to Trellis. A lot of this has been shared by Ben in terms of the, the grounding principles of it. The first is you wanna initialize an output space by launching an MVP agent that is informed by some product priors and some product expectations, but the goal is really to collect a lot of

[16:11](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=971s) user data. The second step is once you've unders-- Once you have all this user data, you wanna correctly classify these into intents based on usage patterns. The goal is you wanna understand exactly why people are sticking to your product and what they're using in your product, especially when it's a conversational, open-ended AI agent experience.

[16:28](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=988s) The third step is converting these intents into dedicated semi-deme- semi-deterministic workflows. A workflow is a predefined set of steps that allows you to achieve a certain output. The goal is you want these workflows to be broad enough to be useful for many possibilities, but narrow enough to be reliable.

[16:45](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1005s) After you have your workflows, you wanna prioritize them by some scoring mechanism. This has to be something that's tied to your company's KPIs. Um, and finally, you wanna analyze these workflows w- from within.

[16:54](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1014s) You wanna understand the failure patterns within them. You wanna understand the sub-intents, and you wanna keep recursing from there, which is what step six involves.

[17:02](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1022s) A quick note on prioritization. There's a simple and naive way to do it, which is volume only. This involves focusing on the workflows that have the most volume. However, this leaves a lot of room on the table for improving general pr- satisfaction across your product.

[17:15](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1035s) A more recommended approach is volume times negative sentiment score. In this, we try to score the ex- the expected lift we'd like to get by focusing on a workflow that might be generating a lot of negative satisfaction on your product.

[17:28](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1048s) An even more informed score is negative sentiment times volume times estimated achievable delta times some strategic relevance. The idea of estimated achievable delta is-- comes down to you coming up with a way to score the actual achievable delta you can gain from working on that workflow and improving the product.

[17:44](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1064s) If you're gonna need to train a foundation model to improve something, its achievable delta is probably near zero, depending on the kind of company you are.

[17:51](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1071s) All in all, the goal is once you have these intents identified, you can build structured workflows where each workflow is self-attributable, deterministic, and is self-bound. Which means-- Which, which allows your teams to move much more quickly, because when you, when you, uh, improve a specific workflow,

[18:09](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1089s) all those changes are contained and self-accountable to that one workflow instead of spilling over into other workflows. This allows your team to move more reliably.

[18:17](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1097s) And, uh, while we have a few more seconds, you can continue to further refine this process, going deeper d- and deeper into all your workflows. And at the end of that, you create magic, which is engineered, repeatable, testable, and attributable, but not accidental.

[18:29](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1109s) If you'd like to read more about this, feel free to scan this QR code to read about our blog post on the Trellis framework.

[18:35](https://www.youtube.com/watch?v=eSvXbb2EBYc&t=1115s) Thank you for having me. [outro music]
