# Beyond the Consensus: Navigating AI’s Frontier in 2025

Grace Isford, Lux Capital

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/HS5a8VIKsvA-beyond-consensus-navigating-ais-frontier-in-2025) · [Video](https://www.youtube.com/watch?v=HS5a8VIKsvA)

## Beyond the Consensus: Navigating AI’s Frontier in 2025

Grace Isford traces the gap between model progress and dependable agents, using a constrained flight search to explain cumulative errors, personalized evaluations, and better product design.

### How do you navigate a frontier that keeps moving?

[0:21](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=21s)

How do you decide what to build when the AI frontier moves faster than the consensus about it? At the [AI Engineer Summit 2025](https://ai.engineer/summit/2025), Grace Isford approaches that question as a partner at Lux Capital, investing in technologies before their usefulness is widely understood. The portfolio examples establish the breadth of that perspective: Hugging Face provides a GitHub-like home for machine learning; Together AI supplies an open-source AI cloud; Physical Intelligence develops a robotics software brain; and Tokyo-based Sakana AI explores nature-inspired evolutionary algorithms. Isford also points to Sakana’s AI CUDA Engineer, which she says launched the preceding night.

New York is part of this investment thesis, not simply the conference location. Isford says Lux was founded there, made its first AI investment there in 2013, and has a majority of its AI portfolio headquartered in the city or operating a major hub there. The combination of research and engineering talent with Fortune 500 customers creates an opportunity to connect frontier capabilities with organizations that can use them.

### More models, more reasoning, more infrastructure

[2:06](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=126s)

Starting with **Stable Diffusion** in August 2022, Isford’s timeline depicts a rapidly steepening release curve. She describes acceleration across the following two and a half years, especially the preceding eighteen months. Development is also spreading beyond OpenAI and Anthropic to xAI, Mistral, and DeepSeek. The **Grok** launch is one recent example of a broader shift: more organizations are producing capable models while improving computational efficiency.

![Slide titled “AI Progress Curve is exponential,” with model names arranged by year and a red curve rising steeply toward 2025.](https://ai.engineer/api/talk-essay-media/6d5085f0568b0df64d8390b0e6aa4db5b627ebd6ece684e61e138ed0d90a2d14)

AI progress depicted as an exponential curve alongside a timeline of model releases.

Early 2025 adds infrastructure commitments and public attention to that technical momentum. The [Stargate announcement](https://openai.com/index/announcing-the-stargate-project/) proposes $500 billion of investment over four years; it is a plan, not completed expenditure. Although Isford groups the US government with the participating companies, the announcement distinguishes government support from the initial corporate equity funders: SoftBank, OpenAI, Oracle, and MGX. She also highlights **OpenAI o3** and its [ARC-AGI preview result](https://arcprize.org/blog/oai-o3-pub-breakthrough). Her description of exceeding human performance needs a narrower reading: this was a benchmark-specific preview evaluation with substantial test-time compute, which ARC Prize characterized as approaching human-level performance, not establishing general human superiority. The subsequently released o3 was a different system from that tested preview.

Next comes **DeepSeek R1**. Isford recounts its launch, the accompanying decline in NVIDIA shares, and DeepSeek’s rise to the top of the App Store. The France AI Summit and Macron’s initiative complete her picture of renewed international participation: the frontier is expanding across model developers, infrastructure providers, and governments.

### Favorable conditions do not establish autonomy

[3:35](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=215s)

These developments create what Isford calls a perfect storm for agents. Reasoning models—**OpenAI o1**, o3, R1, and Grok’s latest reasoning model—suggest new capabilities, including performance she describes as surpassing humans on some tasks. **Test-time compute** is a central mechanism: a system spends additional computation during inference rather than only during training, giving it more opportunity to work through a problem before answering.

Engineering efficiency supplies another part of the argument. Isford praises DeepSeek’s hardware and engineering work without resolving the dispute over its training cost. Cheaper inference and hardware make more agent activity economically plausible, while DeepSeek and **Llama** narrow the gap between open and closed models. Data-center investment in the United States, Europe, and Japan—including efforts involving SoftBank and NVIDIA—provides additional capacity.

Yet the conditions for agents are not the same as agents that reliably work. Isford sees momentum without the decisive breakthrough. Her definition sets a demanding bar: **an agent is a fully autonomous system in which LLMs direct their own actions**. That definition turns attention from the quality of an individual response to whether a system can carry an entire task through successfully.

### An ordinary flight request becomes a constraint problem

[5:23](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=323s)

Consider booking a Monday flight from New York to San Francisco. Isford tries this with [OpenAI Operator](https://openai.com/index/introducing-operator/), using its early-2025 browser-based research preview. That historical experience precedes Operator’s later integration into ChatGPT agent. What sounds like a simple booking instruction contains several interacting requirements.

| Constraint | Requested outcome |
|---|---|
| Departure | After 3 p.m. Monday; avoid rush-hour traffic |
| Airline | United, JetBlue, or American |
| Upgrade | Preserve an opportunity to upgrade from economy |
| Budget | Below $500 under the work expense policy |
| Seat | Aisle, not too close to the bathroom |
| Arrival | Before midnight |

The agent must satisfy the combination, not merely locate an available flight. A departure time can meet the stated time window while still making the trip to the airport impractical.

On the first attempt, Operator goes to Kayak and finds no flight. Isford says it does not even appear to look for United or American. The slide’s completed annotation makes those two failures explicit: the search misses preferred airlines and produces no result.

![“The NYC-SFO Travel Problem” slide with a Kayak screenshot and a red callout reading “First try: Kayak,” “Missed United or American,” and “Didn't find flight.”](https://ai.engineer/api/talk-essay-media/41f371567cfd127fa865f0512c082c9086385a5c007c5e8f5c4d85a1f4ee75c7)

The first Kayak attempt fails to find a flight and misses preferred airlines.

On the second attempt, Operator uses Skyscanner and finds a flight departing JFK at 5:30 p.m. That result conflicts with the request to avoid rush-hour traffic. Isford also cannot select her seat. The search has progressed from no result to a result, but it still has not completed the requested task.

### Small failures extend beyond hallucinations

[6:43](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=403s)

Hallucinations and fabricated answers receive plenty of attention, but this booking attempt exposes a wider problem: small errors accumulate across decisions, integrations, practical judgment, and preferences. Isford offers a non-exhaustive taxonomy. **Decision error** means choosing the wrong fact—for example, interpreting San Francisco as the city in Peru rather than California. Overthinking or exaggeration can also push a decision off course. The wrong destination is a hypothetical illustration, not an outcome of her demonstrated search.

**Implementation error** concerns access and integration. During the Skyscanner attempt, Isford has to complete a CAPTCHA, interrupting the flow. Operator’s original design explicitly required human takeover for CAPTCHAs, making this a boundary of its autonomy rather than necessarily a malfunction. Losing access to a critical database would create a related operational problem: the agent could know what it should do but lack the access needed to continue.

**Heuristic error** means applying the wrong practical criteria. The flight search does not allow adequately for reaching JFK, and it never asks whether Isford is starting from Manhattan, Brooklyn, or somewhere else. Without that information, a seemingly acceptable departure time can still be a bad recommendation.

**Taste error** means missing a personal preference. Isford does not like flying on Boeing 737 MAX aircraft, but she did not include that constraint in the original prompt. An agent could satisfy every explicit requirement and still offer a flight she would refuse to board. This is a different problem from misreading an instruction: the relevant information was never supplied.

A fifth issue is the **perfection paradox**. Capabilities that initially seem extraordinary quickly become the baseline for expectations. Users grow impatient when o1 thinks too long or Operator moves at human speed. Even getting the first attempt right is insufficient if subsequent attempts are inconsistent. Reliability, latency, and the user’s expectations all shape whether the experience feels successful.

### Fifty steps change what “accurate” means

[8:47](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=527s)

Isford illustrates the cumulative effect with two agents that begin at 99% and 95% accuracy per step. Both sound strong in isolation. Across fifty consecutive steps, however, the chance of completing every step correctly can fall sharply.

One mathematical interpretation assumes constant, independent per-step success and requires every step to succeed:

```math
\begin{aligned}
P(\text{all steps succeed}) &= p^n \\
0.99^{50} &\approx 60.5\% \\
0.95^{50} &\approx 7.7\%
\end{aligned}
```

Under those assumptions, the 99% agent succeeds across all fifty steps about 60.5% of the time, versus 7.7% for the 95% agent—a gap of about 52.8 percentage points. This makes precise the rough separation Isford describes; the recording does not specify the probability assumptions or present an empirical trial.

The engineering target is therefore **consistent completion of the whole task**, not simply a high rate of correct individual actions. Booking a flight combines enough decisions and dependencies to expose this difference. Multi-step, multi-agent systems can amplify it further. Isford’s response is five strategies: curate the data, evaluate the actual outcome, contain failures with scaffolding, understand the workflow, and build richer multimodal experiences.

### Curate what the agent needs—and what it learns

[10:04](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=604s)

The first strategy is **data curation**. An agent’s inputs are messy, unstructured, and scattered across silos. They extend beyond web pages and text to designs, images, video, audio, warehouse sensors, and data generated by the agent itself. Curation must cover proprietary source material, agent outputs, and the workflow data used for quality control.

That work should continue after launch. Isford proposes designing a data feedback loop from the beginning, so product usage can improve the system rather than disappear into an activity log. For the travel agent, the initial curated dataset would contain her airline preferences and avoidance of the 737 MAX. Subsequent bookings would supply additional evidence about what she accepts, rejects, or changes. Feeding that information back into the preference dataset could make future recommendations more personal. It is a proposed improvement to the booking system, not something her Operator attempt already accomplished.

### Evaluate the outcome for the person using it

[11:22](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=682s)

The second strategy is **evaluation**: collect model responses and determine what counts as correct. In verifiable tasks, a clear answer supplies a strong signal. Isford points to **Grok 3** math and science benchmarks as examples of this style of evaluation. But whether she will like a particular airplane seat is not reducible to the same kind of answer key. That requires signals about her preferences.

The deep research products from OpenAI, Perplexity, and **Gemini** illustrate why the task and the user matter. A product that works well for everyday research may not be the best fit for venture-capital market research or scientific and academic work. Evaluation must capture those differences rather than assume a single ordering applies to everyone. Collecting human preferences makes it possible to assess the outcome against the user’s intended purpose. Sometimes directly trying an agent on that purpose reveals more than a leaderboard number does.

### Stop a local error from becoming a system failure

[12:43](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=763s)

The third strategy is **scaffolding**: the surrounding system should prevent one failure from cascading into others. Isford cites Lux portfolio company Ramp, reporting that its infrastructure logic contains failures in new applied-AI features so they do not propagate across the agentic system or production infrastructure. The key property is containment; she does not specify Ramp’s implementation.

A compound system can coordinate components and bring a human back into the loop when necessary. Stronger reasoning agents introduce further possibilities: an agent might recognize a mistake and correct its path, or recognize uncertainty and interrupt execution before continuing. In the travel example, a checkpoint could verify the airport timing and redirect the booking agent before an unsuitable flight becomes a completed purchase. The scaffold supplies an opportunity to recover instead of allowing each earlier choice to dictate the next one.

### Shared models make workflow understanding more valuable

[13:41](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=821s)

The fourth strategy moves from reliability to **user experience**. Isford views foundation models as rapidly depreciating assets: when competing applications can access the same underlying capabilities, model access alone offers little durable differentiation. A GPT wrapper can still be a valuable product if it deeply understands the user’s workflow and makes human-machine collaboration effective.

Her examples identify different ways that understanding appears in a product:

- **Clarify the objective:** Deep research asks questions before proceeding, establishing what the user is actually trying to accomplish.
- **Anticipate the next action:** Windsurf from Codeium reflects the developer’s workflow and helps predict the next step.
- **Fit the existing environment:** Harvey integrates into legacy legal systems, which Isford presents as a route to practical value for lawyers.

These are interaction and integration decisions, not merely differences in the text a model generates.

The same distinction applies to coding, customer support, and sales: applications using shared models compete through product quality and UX. Isford is especially interested in domains where proprietary data meets specialized work—robotics, hardware, defense, manufacturing, and life sciences. Understanding how a biologist, defense contractor, or chemist works can determine whether an AI capability becomes a useful product.

### Build beyond the chatbot

[15:19](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=919s)

The fifth strategy is to build **multimodally**, not merely combine multiple models. Isford’s ambition is a dramatically more personalized experience, rather than a measured improvement claimed for an existing system. The chatbot is only one possible interface. Vision, hearing, and increasingly natural voice can give agents other ways to receive context and interact with people.

She extends the idea beyond familiar media. Lux investment [Osmo](https://www.osmo.ai/blog/update-scent-teleportation-we-did-it) works on digitizing smell. Touch and robotics raise questions about embodiment, while memory could let an assistant know a person more deeply over time. Together, these directions change both the information available to an agent and how a person might experience its help.

Isford suggests that a sufficiently novel product experience can exceed expectations even while the underlying agent remains inconsistent. Her concrete example is Lux portfolio company tldraw: a visual canvas where AI can be expressed through brushstrokes. She describes tldraw Computer as combining models in tandem without requiring the user to think about the large language model operating underneath. The visible canvas of connected boxes and drawn strokes illustrates an interface organized around visual work instead of a chat transcript.

![Stage view showing a slide titled “5 Building multimodally,” with bullets about sensory capabilities and personalized experiences beside a canvas of connected boxes and drawn strokes.](https://ai.engineer/api/talk-essay-media/d1ddc7d4a2f12019c1d5aec02f9e846466ded78fb444135f013d0133bb92211b)

Building multimodally through sensory interfaces and a visual canvas.

The favorable conditions for agents are real, but dependable autonomy will not arrive overnight. Wrong answers, missing preferences, unsuitable criteria, and disappointed expectations accumulate. Data curation, evaluations, and scaffolding address those failures; UX and multimodality determine what the resulting product enables people to do. Isford’s closing challenge is to pursue both: make the system more dependable while imagining a better way for the user to work.

## Resources

- [Introducing Operator](https://openai.com/index/introducing-operator/) — Operator's original research-preview announcement explains browser interaction, human takeover and safety limits.
- [ARC Prize's evaluation of preview o3](https://arcprize.org/blog/oai-o3-pub-breakthrough) — The benchmark organizer reports preview o3's ARC-AGI-1 results, compute conditions and limitations.
- [The Stargate Project announcement](https://openai.com/index/announcing-the-stargate-project/) — The original announcement sets out the proposed investment timeline and participating companies.
- [Sakana's later CUDA benchmarking and verification report](https://sakana.ai/ai-cuda-engineer/) — A September 2025 account of robust-kbench and an agentic workflow for generating, verifying and optimizing CUDA kernels.
- [How Osmo digitizes and reconstructs scent](https://www.osmo.ai/blog/update-scent-teleportation-we-did-it) — Osmo describes its laboratory process for measuring scent molecules and reconstructing a fragrance.

## Slides

### Lux Capital AI portfolio

Lux Capital says it has partnered early with AI companies including Hugging Face, Together AI and Sakana AI. The slide lists its 8th fund and $5B AUM.

[0:45](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=45s) [Lux Capital slide stating eighth fund and $5 billion AUM, alongside logos of AI portfolio companies.](https://assets.ai.engineer/sites/aiecode2025/sha256/1cfad15f97d6e948c84f4e8c633864250672173fe84d1acb42ef4b97c91e7a47.webp)

LU+
8th fund, $5B AUM

Hugging Face
mosaicML
Modal
TOGETHER
sakana ai
tldraw
runway
databricks
π
Maven AGI
osmo

[1:12](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=72s) [Lux slide showing its eighth fund, $5 billion in assets under management, and AI company logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/015a6c5db3122f987b7d39c197cea5cc9672119519e9235711b5169ab92680ba.webp)

LU+
8th fund, $5B AUM

Hugging Face
mosaicML
Modal
TOGETHER
sakana ai
tldraw
runway
databricks
π
Maven AGI
osmo

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### 2025 off to a wild start

The ARC-AGI chart plots tuned o3 at 76% and 88%, above o1 scores but at higher cost per task, alongside headlines on Stargate, DeepSeek and France’s AI investment.

[2:50](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=170s) [Headlines about AI investment and DeepSeek surround an ARC-AGI performance chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5f019303a586c04a4c7e0a6a1d97f5bcc86ae7bed65ba6528fc77480eacbf90.webp)

2025 off to a wild start

Trump Announces $100 Billion A.I. Initiative
OpenAI, Oracle and SoftBank formed a new joint venture called Stargate to invest in data centers, building on major U.S. investments in the technology.

DEEPSEEK AI SHOCKS US TECH,
NVIDIA LOSES RECORD $600 BN

O SERIES PERFORMANCE / ARC-AGI SEMI-PRIVATE EVAL
Y-axis: Semi-Private Score (%)
Y-axis ticks: 0%, 25%, 50%, 75%, 100%
X-axis: Cost Per Task
X-axis ticks: $1.0, $10.0, $100.0, $1,000.0
Chart labels: KAGGLE SOTA; AVG. MTURKER; STEM GRAD; O1-MINI; O1 PREVIEW; O1 LOW; O1 MED; O1 HIGH; ARC-AGI-TUNED O3 LOW; ARC-AGI-TUNED O3 HIGH
Visible scores: 7.80%, 13.33%, 25%, 31%, 32%, 76%, 88%

Macron unveils $112B AI investment package, France’s answer to US’ Stargate

Image Credit: NYT, ARC-AGI, ET, Techcrunch

### Perfect Storm for AI Agents in 2025

Test-time compute applies more compute at inference to improve performance. The slide also links reasoning models to capability, engineering and hardware optimizations to efficiency, and infrastructure spending to power.

[3:38](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=218s) [Four factors for AI agents: reasoning models, test-time compute, engineering and hardware optimization, and infrastructure spending.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3f1d286163502ba67485633c405b8a7b58cf3860303d35fa621a5d0d8c99b74.webp)

Perfect Storm for AI Agents in 2025

New reasoning models → more capability
Rise of test-time compute → more performance
Eng & hardware optimizations → more efficiency
Billions of infrastructure spend → more power

[3:47](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=227s) [Four factors in AI agent progress: reasoning models, test-time compute, engineering and hardware optimizations, and infrastructure spending.](https://assets.ai.engineer/sites/aiecode2025/sha256/a20cf43b16955f75d1f4093cc723683505b0b065fb410d7416bf5dce4ddec522.webp)

Perfect Storm for AI Agents in 2025

New reasoning models → more capability
Rise of test-time compute → more performance
Eng & hardware optimizations → more efficiency
Billions of infrastructure spend → more power

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### The NYC–SFO Travel Problem

The Operator request seeks a flight after 3 PM while avoiding rush-hour airport commutes over an hour, staying within $500, arriving by midnight and optimizing for an upgrade.

[5:27](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=327s) [An Operator prompt describes a flight-booking request with timing, airline, budget, traffic, and seating constraints.](https://assets.ai.engineer/sites/aiecode2025/sha256/e13ba93111f85fbd952fad531265927e0c42097fe84dcd59b00db55bef927af9.webp)

The NYC-SFO Travel Problem

Operator

Book a one-way flight on Monday, February 24, from NYC to San Francisco, departing after 3 PM after team meetings from LGA, EWR or JFK. I prefer flying American, United, or JetBlue. I would like to avoid rush hour traffic where my estimated commute time given traffic would be greater than 1 hour. I am JetBlue Mosaic, United Premier 1K, and American Platinum Pro and want to optimize for an upgrade from economy. I would like an aisle seat, not next to the bathroom but as far forward in the plane as possible. My budget is $500 and I don't want to arrive after midnight so I can get a good sleep.

Image Credit: OpenAI

[5:52](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=352s) [Operator prompt requesting a New York to San Francisco flight with timing, airline, seating, traffic, and budget preferences.](https://assets.ai.engineer/sites/aiecode2025/sha256/c1ed3cbb9f917a46385fe63ee5eb497a4a795c7897989c7a58cbdb519759fe8f.webp)

The NYC-SFO Travel Problem

Operator

Book a one-way flight on Monday, February 24, from NYC to San Francisco, departing after 3 PM after team meetings from LGA, EWR or JFK. I prefer flying American, United, or JetBlue. I would like to avoid rush hour traffic where my estimated commute time given traffic would be greater than 1 hour. I am JetBlue Mosaic, United Premier 1K, and American Platinum Pro and want to optimize for an upgrade from economy. I would like an aisle seat, not next to the bathroom but as far forward in the plane as possible. My budget is $500 and I don't want to arrive after midnight so I can get a good sleep.

Image Credit: OpenAI

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Kayak Search Result

Operator’s Kayak search returned no matching results. It reported no JetBlue flights after 3 PM from JFK or Newark on February 24; the speaker says it did not appear to check United or American.

[6:09](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=369s) [Kayak shows no matching results, followed by an Operator response saying it could not find the requested JetBlue flights.](https://assets.ai.engineer/sites/aiecode2025/sha256/911cae49d0277ec1f6fd38b7555d247ab12b09abb7cd5f6fd0437b67799bdfe4.webp)

No matching results found.
Try adding nearby airports to see more results.
Show nearby airports

Try removing some other filters to see more results.
Book on KAYAK ×
Airlines ×
Layover ×
Landing ×
Takeoff ×
Clear all

I couldn't find any JetBlue flights departing after 3 PM from JFK or Newark on February 24th. Would you like to consider flights from other airlines or adjust your preferences?

Interface text:
KAYAK
One-way
JFK, EWR
San Fran... ×
Mon 2/24
1 adult
Economy
Sign in
Ask KAYAK
0 of 413 flights
Stops
Airports
Reset
Takeoff Airports
JFK: John F Kennedy Intl
EWR: Newark
LGA: LaGuardia
San Francisco
SFO: San Francisco
OAK: Oakland
Fee Assistant
Carry-on bag
Checked bag
Book on KAYAK
Times
Compare vs. KAYAK
Compare all
Share feedback?
KAYAK makes it easy to find what I am looking for
1 2 3 4 5 6 7
Strongly disagree
Strongly agree

### The NYC–SFO Travel Problem: Second Try

The Skyscanner retry found flights, but the slide flags a CAPTCHA, Delta’s inclusion, no aisle seat, and more than an hour of traffic.

[6:28](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=388s) [A flight recommendation lists three options, alongside a callout noting a CAPTCHA, an unpreferred airline, no aisle seat, and traffic exceeding one hour.](https://assets.ai.engineer/sites/aiecode2025/sha256/accc3d061e22864283944dd86c648dadf4384e7902d615c09fd5f3a4dddeae0c.webp)

The NYC-SFO Travel Problem

I've found several flights from New York to San Francisco on February 24th that meet your criteria:

1. JetBlue: Departs JFK at 5:29 PM, arrives SFO at 8:17 PM, priced at $239.
2. American Airlines: Departs JFK at 5:00 PM, arrives SFO at 8:54 PM, priced at $244.
3. Delta: Departs JFK at 5:05 PM, arrives SFO at 8:54 PM, priced at $244.

The JetBlue flight aligns with your preferences and offers the best potential for upgrades and preferred seating due to your JetBlue Mosaic status. Would you like to proceed with this option?

Second try: Skyscanner
• CAPTCHA needed
• Included Delta
• No aisle seat
• >1 hour traffic

### Cumulative errors add up

A travel agent can accumulate wrong facts, access problems, criteria that ignore traffic, and missed preferences such as the speaker’s aversion to the Boeing 737 MAX.

[8:01](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=481s) [Decision, implementation, heuristic, and taste errors are paired with examples of incorrect facts, access, criteria, and personal preferences.](https://assets.ai.engineer/sites/aiecode2025/sha256/c2917983122e2f4233d13a033e88a84dc9947caea92779e57eb53462ba284697.webp)

Cumulative errors add up

Decision Error
wrong fact
(eg. “San Francisco, Peru” vs “San Francisco, CA”)

Implementation Error
wrong access/integration
(eg. CAPTCHA needed)

Heuristic Error
wrong criteria
(eg. not accounting for traffic)

Taste Error
wrong personal preferences
(eg. Boeing 737 MAX for a spooked flyer)

### Agents = Complex systems with compounding errors

Across 50 consecutive steps, the 99% agent falls to about 60%, while the 95% agent falls below 10%. The chart marks a 52.8 percentage-point gap.

[8:50](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=530s) [A chart compares 99% and 95% agent accuracy over 50 steps, showing a 52.8 percentage-point gap at the end.](https://assets.ai.engineer/sites/aiecode2025/sha256/199d1a5d21e53062c0551026bf7415e8ed35e282fd555080ad985516e8be6eba.webp)

Agents = Complex systems with compounding errors

Y-axis ticks: 0%, 20%, 40%, 60%, 80%, 100%
X-axis ticks: 0, 10, 20, 30, 40, 50
Legend: 99% agent; 95% agent
52.8%

[9:03](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=543s) [A graph compares 99% and 95% agent accuracy over 50 steps, showing a 52.8 percentage-point gap.](https://assets.ai.engineer/sites/aiecode2025/sha256/857f306533f9658693b768dd13a8e084c4f149a7d6ef80cb54e271d9075656c5.webp)

Agents = Complex systems with compounding errors

Y-axis ticks: 0%, 20%, 40%, 60%, 80%, 100%
X-axis ticks: 10, 20, 30, 40, 50
Legend: 99% agent; 95% agent
52.8%

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### How to optimize a complex agent?

A flight-booking agent takes preferences, heuristics, implementation requirements and information as inputs to select a winning flight. The challenge is making the right decision consistently and reliably.

[9:39](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=579s) [Flight preferences, heuristics, implementation requirements and information feed an AI agent that selects a winning flight.](https://assets.ai.engineer/sites/aiecode2025/sha256/0dbee02c5d603f976d87d2bfb99c8beeef71293c77a92ca1ce9e8ca8fa574812.webp)

How to optimize a complex agent?

Data integration
Not near bathroom
Maximize upgrade probability
System authentication
Minimize traffic
Not fly 737 MAX
Arrive 30 mins before boarding
JetBlue, United, American
API access
Land no later than midnight
San Francisco, CA

AI Agent → WINNING FLIGHT

Taste
Heuristic
Implementation
Information

### There’s hope!

The speaker introduces five strategies to help mitigate cumulative errors when building AI agents.

[9:48](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=588s) [The words “There’s hope!” appear in red on a dark slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/d68b5dc5911d79b2ea2024fadb0f2833088242eaf409101034e00683bce809bb.webp)

There’s hope!

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Data Curation

Agents draw on data spread across silos, from sensors and audio to their own outputs. Curation makes that data more effective; the slide frames rejection sampling as quality control.

[10:07](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=607s) [Data curation slide describing quality control, data advantages and error mitigation, alongside a diagram of data types.](https://assets.ai.engineer/sites/aiecode2025/sha256/363832c69e8dfd2900ec4723a4641b7500acc21677f8a61922600860dc86346c.webp)

1 Data Curation

• How do we make sure an AI agent has information it needs?
• Data is messy, unstructured & across silos
• Curation makes it more effective
  ◦ Rejection sampling = “quality control”
• Data advantage differentiates agent performance & amplifies with scale
  ◦ Ex: Comprehensive dataset of travel preferences, agent workflow data
• Mitigates decision + heuristic errors

Image data
Sensor data
Audio data
Agent data
Machine data
Biological data
Video data

### Importance of Evals

Evals can score clear answers in math and science, but a seat recommendation needs signals about personal preferences. The slide also calls for evaluating whole systems, not just models.

[11:20](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=680s) [Evaluations slide covering verifiable and non-verifiable domains, system evaluations and a benchmark chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/4af2e8bcc3e03c782ac93cb20a22390af403c999d5cc9c674f6097fcce86ed62.webp)

2 Importance of Evals

• How do we collect & measure a model’s response?
• How to choose the practically “right” answer?
• Evals in verifiable domains - collect signal!
• Evals in non-verifiable domains?
  ◦ “Is this a good plane seat based on my preferences?”
• Evals for systems (vs models)
• Mitigates decision + heuristic + taste errors

Benchmarks
Y-axis: Score
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70, 80
X-axis: x1
Categories: Math(AIME’24), Science(GPQA), Coding(LCB Oct-Feb)
Legend: Grok-3, Grok-3 mini, Gemini-2 Pro, DeepSeek-V3, Claude 3.5 Sonnet, GPT-4o
Value labels, in legend order:
Math(AIME’24): 52, 40, 36, 39, 16, 9
Science(GPQA): 75, 65, 65, 59, 65, 50
Coding(LCB Oct-Feb): 57, 41, 36, 40, 36, 34

Image Credit: Grok3

### Scaffolding Systems

Scaffolding links an LLM with retrieval, tools and memory. The speaker describes infrastructure that keeps a failed AI feature from cascading across an agentic system or production infrastructure.

[12:43](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=763s) [Scaffolding systems slide covering robust logic, human oversight and self-reflection, with a diagram connecting an LLM to retrieval, tools and memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/afd568031796a724f10b1c1beaad4bfce8344a0f0c5ca659baa6ef4031c00253.webp)

3 Scaffolding Systems

• How do we ensure when one error occurs it doesn’t have a cascading effect?
• Robust, well-developed logic & tools
  ○ Prompt chaining, routing, orchestration, observability, code interpreter
  ○ Human-in-Loop flows
• Self-reflection for reasoning models
• Mitigates decision + implementation errors

In → LLM → Out
QUERY/RESULTS → Retrieval
CALL/RESPONSE → Tools
READ/WRITE → Memory

Image Credit: Anthropic

[13:08](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=788s) [Slide describing safeguards against cascading errors, including orchestration, human oversight and self-reflection, alongside an LLM diagram with retrieval, tools and memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/9786cad7d310a2905a2747d016bd43459d40ddee5d2d31e51b6a098aa45f0eef.webp)

3 Scaffolding Systems

• How do we ensure when one error occurs it doesn’t have a cascading effect?
• Robust, well-developed logic & tools
  ○ Prompt chaining, routing, orchestration, observability, code interpreter
  ○ Human-in-Loop flows
• Self-reflection for reasoning models
• Mitigates decision + implementation errors

In → LLM → Out
QUERY/RESULTS → Retrieval
CALL/RESPONSE → Tools
READ/WRITE → Memory

Image Credit: Anthropic

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### UX is the Moat that Matters

The speaker argues that product UX can differentiate AI apps as foundation models depreciate: Deep Research asks clarifying questions, Windsurf anticipates developers, and Harvey fits legal workflows.

[13:44](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=824s) [Slide arguing for user-centered AI products, with examples involving Deep Research, Windsurf and Harvey.](https://assets.ai.engineer/sites/aiecode2025/sha256/de515c6bd75a71529d63a386b83f3bba4584639130a8f1ef52b965d421e15673.webp)

4 UX is the Moat that Matters

• How do we know the user better & become a better copilot?
• Foundation models = fastest depreciating asset
• “GPT Wrappers” are cool
  ◦ Product/UX > owning model weights
  ◦ Elegant & enjoyable AI collaboration
• Reimagines UX & mitigates heuristic + taste errors

Deep Research asking clarifying questions
Windsurf understanding the UX + psyche of developer
Harvey seamlessly integrating LLMs to legal workflows

[14:11](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=851s) [UX principles and examples from Deep Research, Windsurf, and Harvey.](https://assets.ai.engineer/sites/aiecode2025/sha256/30c4640fa45306e2adb990673dabb41b84bec1b3449c5eb9531e4183f856f800.webp)

4 UX is the Moat that Matters

• How do we know the user better & become a better copilot?
• Foundation models = fastest depreciating asset
• “GPT Wrappers” are cool
  ◦ Product/UX > owning model weights
  ◦ Elegant & enjoyable AI collaboration
• Reimagines UX & mitigates heuristic + taste errors

Deep Research asking clarifying questions
Windsurf understanding the UX + psyche of developer
Harvey seamlessly integrating LLMs to legal workflows

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### AI Applications and New Frontiers

Coding, support and sales apps share foundation models, making UX and product quality important differentiators. The next frontier combines proprietary data and workflow knowledge in robotics, defense, manufacturing and sciences.

[14:42](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=882s) [Coding, customer support, and sales applications share foundation models; robotics, manufacturing, and sciences are identified as new frontiers.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea85ffb41d57b63cf2e9442a36906308676ab75f17c2de8ac75db532fa7ab1b1.webp)

4 UX is the Moat that Matters

All using the same models:

CODING
CURSOR
codeium
Cognition

CUSTOMER SUPPORT
DECAGON
SIERRA
Maven AGI

SALES
rox
clay
11x

GPT 4o, Claude Sonnet 3.5, Llama 3, DeepSeek R1

The new AI frontier:
ROBOTICS & HARDWARE
DEFENSE & MANUFACTURING
BIOLOGY, CHEMISTRY & PHYSICS

### Building Multimodally

The slide proposes moving beyond chatbots with vision, audio, speech, smell and robotics. A drawing interface illustrates how new modalities could reshape personalized product experiences.

[15:20](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=920s) [Multimodal AI product ideas include vision, audio, speech, smell, and robotics, alongside a drawing interface example.](https://assets.ai.engineer/sites/aiecode2025/sha256/baefac6574609e0d9a372c5f360bea301af5ae47a95716893bc7c532ee793643.webp)

5 Building multimodally

• How do we get away from the AI chatbot?
• New modalities and reimagined 10x personalized product experiences?
• Giving AI eyes, ears, voice, nose, touch
  ○ Vision, audio, speech, smell, robotics
• Reframes perfection paradox - unlike anything else we’ve seen

draw
diagram
friends
Sticky notes for everyone (and anything)

Image Credit: Tldraw

Interface text:
Page 1
S M L XL
[illegible]%

### Recap

AI agents face cumulative errors in answers, access, criteria and preferences. Data curation, evals and scaffolding can help; UX and multimodality can set agents apart.

[16:53](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=1013s) [Recap of AI agent challenges, mitigation tools, and opportunities in multimodal product experiences.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a73823f6f50d94dd986c68dec2e034c08dad980f68b22677ae7e63ced808164.webp)

Recap

→ Perfect storm for AI agents /= AI agents overnight
→ Wrong answers, wrong access, wrong criteria, wrong preferences, wrong expectations abound
→ Data curation, evals, scaffolding, are tools to mitigate
→ Think bigger - UX, multimodality & innovative product experiences set AI agents apart

## Full transcript

[0:00](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=0s) [on-hold music] Thank you so much, Peter, to swyx, to all of the AI Engineer Summit for having me.

[0:21](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=21s) I am so thrilled to be here. I'm Grace again, a partner at Lux Capital, and it's a pleasure to really kick off this conference and tackle a pretty tough but exciting task, the state of the AI frontier and how we navigate that in twenty twenty-five.

[0:38](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=38s) So a little bit about Lux as we get started here, right? Lux likes to say, "We believe before others understand." We invest in frontier tech ideas that seem crazy, right?

[0:48](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=48s) Uh, and we really like to bring sci-fi to sci-fact. In fact, we've been lucky to partner at the earliest stages with some top AI companies. Hugging Face, I'm sure several folks know it, the GitHub for machine learning.

[1:02](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=62s) Together AI, the open-source AI cloud. Physical Intelligence, that's like a robotics software brain. And Sakana AI, that's a research lab actually in Tokyo, Japan, doing really cool evolutionary nature-inspired algorithms.

[1:16](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=76s) They just launched a pretty cool AI CUDA Engineer last night, so go check it out. Moving a little bit forward as we think about New York City, if I get my clicker working here.

[1:28](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=88s) There we go. Lux is really excited to double down in New York City and AI. Lux was founded in New York City. Our first AI investment was here in twenty thirteen, and a majority of the Lux AI portfolio is actually headquartered here or has a major hub, as you can kinda see in the graph behind me.

[1:46](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=106s) It's also home to many of you, right? State-of-the-art research and engineering leaders and many Fortune five hundred companies, several of whom you're gonna hear from over the next few days.

[1:57](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=117s) We are really bullish on the New York City opportunity, and we're really excited you guys all came to share this opportunity with us.

[2:06](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=126s) So when I was creating this presentation, I went back and looked at the last few years of AI, right? And all the way back really to Stable Diffusion, August twenty twenty-two.

[2:14](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=134s) And wow, I mean, look at this hockey stick, right? The last two and a half years have been crazy. The last eighteen months have been even more exponential. The progress is getting more aggressive, it's getting more impressive, and really it's getting more spread, right?

[2:28](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=148s) It's not just OpenAI and Anthropic publishing these models, it's xAI. We just saw the Grok launch this past week. It's Mistral, it's DeepSeek, it's many, many more. And the models are getting more performant.

[2:40](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=160s) They're also getting more compute efficient. And as we zoom in to the current state of the world in twenty twenty-five, it's off to an even wilder start, right? If you thought the last few years were crazy, twenty twenty-five is even wilder.

[2:55](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=175s) We saw the five hundred billion Stargate Project announced between the US government, OpenAI, SoftBank, and Oracle. We saw o3, OpenAI's o3 right before the start of the year, where they actually exceeded human performance in the ARC-AGI challenge.

[3:10](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=190s) We saw the DeepSeek mania, right? With DeepSeek's R1 model launching earlier this year, sending, you know, NVIDIA shares tumbling down. We also saw DeepSeek go to number one, uh, in the App Store.

[3:24](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=204s) And then of course, just last week, we saw the France AI Summit, where Macron actually launched a whole new AI initiative with France and Europe back in the game.

[3:35](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=215s) So you may be saying, and I think a lot of us are thinking, right, this is the AI agent moment in twenty twenty-five. And I'd go as far as say this is the perfect storm for, for AI agents.

[3:46](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=226s) And frankly, it's easy to see why, right? Uh, several reasoning models starting with OpenAI's o1, then o3, DeepSeek's R1, Grok's latest reasoning model this past week are outperforming human ability and in some cases, even having more capabilities that we've never even seen before.

[4:03](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=243s) We've seen the rise of test-time compute, right? That's more compute applied at inference instead of at training. That's increasing this model performance as well. We've seen further engineering and hardware optimizations, right?

[4:15](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=255s) Whatever you think it costs to actually train that DeepSeek model, you cannot deny it was a feat of engineering and hardware efficiency. Inference is getting cheaper. Hardware is getting cheaper.

[4:27](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=267s) The open source, closed source gap is getting closed between DeepSeek and Llama models getting more and more performant. And of course, billions of infrastructure powering all this data center and compute.

[4:38](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=278s) We just talked about the US Stargate. We talked about Macron and Europe, and also Japan with SoftBank and NVIDIA has been doubling down on their data center efforts. So all this is setting this exciting groundwork for the eponymous name, right, of our conference, Agents at Work, and it really does feel like an exciting moment.

[4:57](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=297s) But in reality, these AI agents aren't really working [chuckles] just yet, right? People are saying it's a perfect storm, and I'm seeing a lot of thunder, I'm seeing a lot of great momentum, but we haven't seen that lightning strike.

[5:08](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=308s) Uh, and everyone I know has different definition of agents, so for the purposes of this presentation, I'm going to define my definition as an AI agent that is a fully autonomous system where LLMs direct their own actions.

[5:23](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=323s) So let's give a little example of what I mean when I say an AI agent isn't working just yet. Here's a seemingly simple query on OpenAI Operator, I'm sure everyone here knows what it is.

[5:33](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=333s) I asked it to book a flight for me from New York to San Francisco on Monday. I'm sure it's also a route and something that many people in this room are familiar with.

[5:41](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=341s) And in reality, it's actually kind of a complex problem, right? I need to leave after three on Monday, but I wanna avoid rush hour traffic. I wanna fly United, JetBlue, or American to maximize my chance of an upgrade from economy.

[5:55](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=355s) I wanna keep it under five hundred dollars to keep under my work expense policy. I also want an aisle seat that's not too close to the bathroom, um, and I wanna get there, you know, before midnight.

[6:05](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=365s) So I put this into OpenAI Operator.

[6:09](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=369s) And the first thing it did with all this information is go to Kayak, which if anyone has booked a flight before, that's a pretty frustrating experience. And unfortunately, it did not find a flight.

[6:18](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=378s) Uh, it wasn't able-- Um, it couldn't find one. It didn't even seem to look for United or American. Second try, try it again. Uh, this is Skyscanner this time, which is slightly better.

[6:27](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=387s) Um, and it did actually find a flight, but it found one that had a lot of traffic. [laughs] Uh, five thirty JFK, for those who live in New York, that is a tough traffic time.

[6:36](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=396s) Um, and ultimately, I also couldn't even pick my seat, so didn't really work out based on my prompts, uh, earlier.

[6:43](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=403s) So what does this all mean, right? Why these AI agents not work? I think we so often talk about hallucinations and fabrications and AI models kind of going sideways.

[6:53](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=413s) We don't talk enough about these tiny cumulative errors that add up, right? Uh, there's a lot of little errors that we see with this whole model, and I'm gonna go through a few.

[7:03](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=423s) It's not an exhaustive list, but it's a sense of some of the things you might run into as you're building these AI agents. First, decision error. It chooses the wrong fact, right?

[7:13](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=433s) I may book a flight with AI, but it may book it to San Francisco, Peru, instead of San Francisco, California. The model could overthink or exaggerate and do a few other things as well.

[7:24](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=444s) Second, implementation error. The wrong access or integration. On the prior slide with my Skyscanner, I actually had to enter CAPTCHA, and that messed up a little bit of the flow.

[7:34](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=454s) You also could get locked out of a critical access to an important database, and ultimately, that AI agent isn't gonna work anymore. Third, heuristic error. The wrong criteria. Unfortunately, the model didn't acknowledge best practice of allowing enough time for JFK.

[7:51](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=471s) In fact, it didn't even ask where I was coming from, Manhattan, Brooklyn, or beyond, and that could really affect the traffic you're gonna get, and ultimately, if I even make that flight at five thirty PM.

[8:01](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=481s) And fourth, taste error. The wrong personal preferences. For those who know me well, I'm actually a pretty spooked flyer, and I do not like flying a Boeing seven three seven MAXes.

[8:11](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=491s) If AI books that, you know, I did not put it in the prompt earlier, but if it did book that, I would be very unhappy, and I would not get on that plane.

[8:21](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=501s) And then there's kind of a fifth, more nebulous error, right? It's a little bit of this perfection paradox, right? We are doing things so magical with AI right now, yet we're getting frustrated when o1 thinks too long or when Operator moves at the speed of a human.

[8:36](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=516s) Even if the agent gets it right on the first try, often they're inconsistent and unreliable, leading to really underwhelming our human expectations about the whole experience.

[8:47](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=527s) Here's another visual of kind of these complex systems where each of these cumulative errors really compound, right? Two simple agents, one that's ninety-nine percent accuracy, one that's ninety-five percent accuracy to start.

[8:59](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=539s) Pretty impressive agents at the beginning. But over fifty consecutive steps, you actually realize a pretty big disparity here. There's actually a fifty percent difference after fifty tasks between the ninety-five and the ninety-nine, and that ninety-nine percent agent's actually down to sixty percent.

[9:14](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=554s) The point here is that something simple like booking a flight is actually really complex in nature when all these tiny cumulative errors add up, and they get even more amplified in a complex multi-agent system with multi-step tasks.

[9:29](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=569s) So how do you, as all these amazing VPs of AI, these leaders of AI in the room, optimize a complex agent, taking into account all of these really difficult queries to consistently and reliably make the right decision?

[9:44](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=584s) The truth is, it's hard. But that hasn't stopped us before, and there is hope. So I thought I would run through some of the best practices that we're seeing building AI agents today and five strategies we can all think about to help mitigate a lot of these cumulative errors.

[10:04](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=604s) Let's dive in. First, data curation. How do we make sure an AI agent has the information it needs? Data is messy. It's unstructured. It's in silos. It's everywhere. It's not just web and text data now too.

[10:19](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=619s) It's design data. It's image data. It's video data. It's audio data. It's the data in your sensors in your warehouse if you're in the manufacturing world. It's even the agent data that your data-- Your agent is producing in real time.

[10:31](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=631s) Think about curating proprietary data, the data the AI d-- Agent generates, and ultimately even the data you're using in your model workflow for quality control. Data is your best asset, and curation is key to making it more effective.

[10:46](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=646s) Data also isn't static anymore. How do you design an agent data flywheel from day one, so every time a user uses the product, it automatically improves in real time and at scale?

[10:59](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=659s) A simple example, back to our flight example, is getting a curated data set of all of Grace's travel preferences, including the seven three seven MAX and all my airline preferences.

[11:09](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=669s) Or even say we run that agent over time and book many flights, how do we recycle back that content and adapt to my own preferences in real time? Second, the importance of evals.

[11:22](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=682s) How do we collect and measure a model's response? How do we choose the correct answer? This has long been important in machine learning and AI and really understanding what's right versus wrong.

[11:33](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=693s) You know, it's pretty simple in verifiable domains where there's a clear yes or no answer, like math, like science. Here's actually the Grok 3 benchmarks just up here, where you saw they did all verifiable benchmarks in math and sciences.

[11:47](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=707s) But how do we set up evaluations for non-verifiable systems where there aren't clear yes or no answers? Like, will Grace like this plane seat based on her preferences? And how do we collect those signals too?

[12:02](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=722s) We also saw in other examples of an eval debate over the weekend with deep research, right? We have an OpenAI deep research product, one from Perplexity, one from Gemini as well.

[12:10](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=730s) And there are multiple versions of the same product. The evals here really depend on the eye of the beholder, right? Which one is better for everyday research versus VC market research versus scientific or academic research?

[12:24](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=744s) We have to keep an eye on collecting those signals. We need to know and collect human preferences, and we need to build evals in a way that is truly personal.

[12:34](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=754s) Sometimes the best eval is just trying out the agent yourself [laughs] and vibes based on your needs with no number or leaderboard telling you what to do.

[12:43](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=763s) Third, scaffolding systems. How do we ensure when one error occurs, it doesn't have a cascading effect throughout the organization? Ramp, a Lux portfolio company, has done a great job with this, and I know Rahul is speaking tomorrow, so go check him out.

[12:55](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=775s) When Ramp launches a new applied AI feature and it fails, there's infrastructure logic to ensure that it doesn't have a cascading effect across the agentic system and also across all of Ramp production infrastructure.

[13:06](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=786s) We can mitigate scaffolding by building a complex compound system of how all these things work together and sometimes even bringing a human back in the loop. For reasoning models, g- this gets even more interesting and important.

[13:18](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=798s) How do we adapt the scaffold to stronger agents that self-heal and grow? An agent that realizes they're wrong and actually tries to correct their own path, or an agent that's not sure, and they need to break execution to get it back on track.

[13:32](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=812s) Back to our travel example again, could we add a checkpoint for this AI agent to verify the timing for traffic or maybe steer it back in the right direction?

[13:41](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=821s) Th- fourth, user experience or UX is the moat that matters, and that's how our AI agents become better copilots. AI apps today are all using the same models. Foundation models are the fastest depreciating asset class on the market right now.

[13:58](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=838s) GPT wrappers are cool. UX really does make a difference for those who reimagine product experiences and really deeply understand the user workflow and really promote that beautiful, elegant human-machine collaboration.

[14:11](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=851s) Right? Here's a few concrete examples. Back to the deep research, right? Asking clarifying questions to make sure it fully got the picture of what I'm trying to accomplish. Like Windsurf from Codeium understanding the UX or the psyche of that developer really on a more fundamental level to predict their next step.

[14:28](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=868s) Like Harvey in the legal world integrating seamlessly with the legacy systems to really create real ROI for a practicing lawyer.

[14:37](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=877s) If you think about all the major AI apps today and categories like coding, like customer support, like sales, these all are using the same models again, right? And it's truly the UX and the product quality that makes any one company stand out.

[14:51](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=891s) At Lux, we're really excited about the new AI frontier. Companies who have proprietary data sources and who know the workflow of their user really well, like robotics, like hardware, like defense and manufacturing, like the life sciences.

[15:06](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=906s) You know, how do we take a company where they take their proprietary data source, they know the workflow of a biologist or a defense contractor or a chemist, and truly create a magical experience for that end user?

[15:19](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=919s) Fifth and finally, how do we build multimodally? You know, we're not just multi-model anymore, we're multimodal. There's new modalities where we can truly reimagine and create a 10X user personalized experience.

[15:34](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=934s) I am so sick and tired of the chatbot as an interface, and I know there's so many more exciting things we can do with our AI agents to really make them more human, right?

[15:43](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=943s) How do we make AI more human? How do we add eyes and ears, nose, a voice? We've seen really incredible improvements in voice over the last year. It's getting pretty scary good.

[15:54](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=954s) Lux actually has an investment in the smell space called Osmo that's digitizing the sense of smell. And what about touch, right? How do we instill a more human feeling and sense of embodiment with robotics?

[16:07](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=967s) I'll go as far to even talk about memories, right? How do we make AI truly personal and know you on a much deeper level than it does today?

[16:16](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=976s) Doing all of this reframes what perfection is to a human, and even if that agent is inconsistent, it's unreliable, the visionary nature of the product exceeds all expectations. It's something new.

[16:27](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=987s) And on the slide behind me, you'll see tldraw. That's an amazing Lux portfolio company, and I think they've done a great job really reimagining the visual canvas, right?

[16:34](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=994s) Implementing AI through brushstrokes. They have a cool thing called tldraw Computer, where you can actually combine a bunch of these cool AI models in tandem and not even know you're working with a large language model in the background.

[16:46](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=1006s) So really strive to build multimodally. So in summary, we tackled a lot today, but we're at the perfect storm today for AI agents. But unfortunately, that lightning hasn't struck yet, and AI agents are not gonna happen overnight.

[17:00](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=1020s) Cumulative errors add up. We see wrong answers, wrong preferences, wrong criteria, all these wrong human expectations that abound when you're building these systems. Data curation, evals, and scaffolding are all tools you can use to help mitigate a lot of these challenges.

[17:15](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=1035s) And really, please think bigger. UX, multimodality, innovative product experience that truly set the workflow and the vision apart. And I'm so excited to see what all of you build and really excited to continue this conversation over the next few days.

[17:31](https://www.youtube.com/watch?v=HS5a8VIKsvA&t=1051s) Thank you guys so much, and look forward to talking to you throughout the conference. Thank you. [upbeat music]
