# Pick Problems That Survive When AI Solves Coding - Ara Khan, Cline

Ara Khan, Cline

[Talk page](https://ai.engineer/talks/iK9gmxXyBR0-pick-problems-that-survive-when-ai-solves) · [Video](https://www.youtube.com/watch?v=iK9gmxXyBR0)

## Pick Problems That Survive When AI Solves Coding

At Cline, Ara Khan gave a coding agent 17 hours to improve the harness around a second model. The failure traces and a slow-talking hardware project point to a way to choose engineering work that grows more useful as models improve.

![A staircase plots the 69/89 baseline, harness changes, and the 79/89 confirmation run.](https://ai.engineer/api/talk-essay-media/e5d3bedaa22c601e4c5f7b65cfbff915bda4fe413058526b3b971ff5f445a819)

Khan’s Terminal-Bench chart tracks changes to one model’s harness from the baseline to the confirmation run.

### What still counts as proof of work?

[0:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=17s)

An engineer used to be able to put a weekend project on GitHub and let the working software speak for itself. What does the same repository tell a hiring manager when a coding agent can build it in an afternoon? Ara Khan, a founding engineer at Cline, takes that question seriously. A project can be useful to its users and still say very little about the judgment it took to build.

He starts with a piece of his own work that a stronger coding model made more interesting. Running **Kimi K3** inside Cline gave his team 69 passes on 89 Terminal-Bench 2.1 tasks. The model developer had reported a higher score. Cline could read the failures and change the software around K3. Could another coding agent do the investigating and the editing, and leave the team with a harness worth keeping?

### Test the work the agent leaves behind

[5:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=300s)

The jump from a chatbot to a coding colleague required tools that could change things outside a conversation. A system that can list a directory, read files, edit them and run commands can try its own fix. Khan remembers building early multi-agent conversations in which much of the apparent collaboration came from the developer wiring messages between classes. Giving agents more people to talk to still left the developer with the hard question of whether any work had been done.

Take a request to set up an MCP server. An agent might have to find the documentation, edit a configuration file, install a dependency and start a process. A confident final message tells you little about whether the client can now use the server. An independent test can open the client and exercise the tool. The transcript then becomes a way to explain a pass or a failure.

Cline used the same separation at benchmark scale. Each of the 89 terminal tasks got an isolated container and a grader. The pass count gave the team a starting point. Tool calls, token use, elapsed time and command output let it distinguish a broken file reader from a working session that exhausted its time allowance. The next decision depended on which explanation recurred.

### Let a second agent read the failures

[11:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=665s)

Khan put **GPT-5.6 Sol** in charge of the outer loop. K3 attempted the tasks through Cline. Sol read the traces, grouped similar failures, proposed a harness change and ran the affected cases again. With a full evaluation taking roughly an hour, a useful lead agent had to remember which hypotheses it had tried and when a change had earned another full run.

The loop detector gave it a particularly good clue. A shell command could appear unchanged across several turns because the agent was polling a background job. The job’s output kept changing. Killing the session for repeating a command threw away evidence of progress that the harness already had. Checking the output as well let healthy tasks continue. A different group of failures came from a provider rate limit ending otherwise viable sessions. Those failures called for waiting and retrying.

The results slide puts both runs on the same denominator. The stock harness passed 69 of 89 tasks (77.5%). The confirmation run passed 79 of 89 (88.8%). In Khan’s published account, that final run cost $49.80. The full 17-hour experiment, including the repeated evaluations and the agent doing the editing, cost roughly $680. Those numbers answer two different questions an engineering lead would ask about doing this again.

The team kept the grader separate, ruled out task-specific shortcuts and reviewed the resulting pull request. Khan also cautions against adapting the harness to every mistake as it happens. A tool change that helps on several unrelated tasks has a better claim to belong in the product.

![The rollout agent acts on its environment while a separate meta-agent changes the harness and evaluates the result.](https://ai.engineer/api/talk-essay-media/529eb69ad03efcca98894f44f458481b0a1a6dad54766f1271fe997540891167)

The inner agent works on a task. A second agent uses the traces to change and evaluate the harness.

### Turn a correction into a test

[14:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=887s)

Khan widens the lens from the harness to the rest of the stack. Training-data selection, post-training, inference kernels and evaluation all offer different ways to make the next run better. Faster inference matters twice in a fixed-budget experiment. It improves the experience of the person waiting for an answer and may buy the team another round of trials.

His database example starts with an ordinary request. Add a column, populate it and put a cache in front. The agent completes the migration but chooses the wrong cache lifetime. A developer tells it which TTL to use, watches it fix the problem and moves on. That short exchange identifies a boundary of the system’s competence much more precisely than a thumbs-down on the whole task.

An engineer can turn that observation into a reproducible case with a check for what happens when the cached value expires. Then it becomes possible to compare a better tool, different instructions or a different model on the behavior that actually failed. A collection of such cases also gives the engineer somewhere to look for regressions after the next change.

![An attempt leads to human correction and then to a verified improvement.](https://ai.engineer/api/talk-essay-media/c488239ed13e26dda3208a4dc0f1f83a5fb4ea75a7cb55650d0a2ebb678ed478)

The slide separates a human correction from the test that verifies an improvement.

### A hardware loop needs a measurement

[18:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1130s)

In a personal project, Khan had a small device with a camera, speaker and microphone. It responded to a question about what it could see, but spoke back too slowly. He connected it to his computer and had a coding agent send requests to the device and time the replies. The device supplied something a second chatbot could not: an end-to-end measurement taken on the hardware that would actually be used.

The overnight experiments could change the firmware, the tool calls, the inference model or the speech-to-text component. Once the computer could put a repeatable question to the device, a change in any one component could be judged by what happened at the speaker. Khan reports the device got faster. The transferable part of the experiment is how he made latency observable across the whole path.

### Choose a problem that benefits from a better model

[21:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1313s)

Khan’s career advice follows from the difference between the weekend repository and the hardware on his desk. A new model release might build the old demo in one shot. That same release gives the engineer with the device a better candidate to test. The apparatus, the latency measurement and the knowledge of what a good interaction feels like all carry forward.

He suggests asking two questions before committing to a long project. If the next model gets much better, which part of the work becomes more valuable? Which part could disappear entirely? Data pipelines that expose real failures, hardware that can test a change and networks that make experiments cheaper all give concrete answers to the first question. Sometimes the honest answer to the second is that it makes sense to wait one release.

The time horizon is a judgment call, and Khan invites disagreement about it. His practical test works on a shorter clock. When an agent can already finish most of an engineer’s daily tasks, look for the correction that still requires that engineer. Can someone capture the failure, tell when it is fixed and give the next attempt better conditions? That is a piece of work a more capable model can help compound.

## Resources

- [Cline’s experiment log and cost breakdown](https://cline.bot/blog/recursive-self-improvement-for-coding-agents) — Ara Khan documents the five harness experiments, failed runs and final confirmation.
- [Terminal-Bench 2.1](https://www.tbench.ai/news/terminal-bench-2-1) — The benchmark authors describe the 89-task suite and its evaluation changes.

## Slides

### Recursively Self-Improvement for Coding Agents

[0:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=0s) [Ara Khan introduces recursive improvement for coding agents and identifies himself as a founding engineer at Cline.](https://assets.ai.engineer/sites/aiecode2025/sha256/f49ded0ac94902550c0bc3b0350b9a4d788909f6b956f0c1a59491b1da1d820b.webp)

Recursively Self-Improvement for Coding Agents
Ara Khan · Founding Engineer, Cline
AI Engineer Summit

### Understand the intent of frontier labs

[0:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=36s) [The first goal is understanding what frontier labs want to build.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e4736e78342775bad90022260d4ec7ad32df6b3b524bf8a0b25d703ea76dad5.webp)

Goal 01 of 03
Understand the intent
of frontier labs

### Career goals for the next few years

[1:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=67s) [The second goal asks engineers to think about their careers as coding models improve.](https://assets.ai.engineer/sites/aiecode2025/sha256/212681b1079e10509b57dddfa59a2a06113b6915f9db8b76957eaca0fa197bca.webp)

Goal 02 of 03
Think about your career goals
over the next few years

### Where does the value go?

[1:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=110s) [The third goal is headed “Escape the permanent underclass.”](https://assets.ai.engineer/sites/aiecode2025/sha256/0bdc4873791c8e5d0291a86498ce06e3741cafd66726a6aa3a6888212165103e.webp)

Goal 03 of 03
Escape the
permanent underclass

### Kimi K3 and Cline on Terminal-Bench 2.1

[3:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=187s) [A staircase plots the 69/89 baseline, harness changes, and the 79/89 confirmation run.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5d3bedaa22c601e4c5f7b65cfbff915bda4fe413058526b3b971ff5f445a819.webp)

Kimi K3 + Cline score on Terminal-Bench 2.1
Eval Score
77.5% · Baseline 69/89
Bounded provider retries
Progress-aware loop detection
File-index request liveness
Safe background-process guidance
86.5% · Frozen full stack 77/89
88.8% · Best run 79/89
Unchanged confirmation
Intermediate step heights are schematic — individual 89-task contributions were not isolated.
Iterative Cline harness improvements

### Self-improving coding agents

[4:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=296s) [The example is a coding agent that improves itself.](https://assets.ai.engineer/sites/aiecode2025/sha256/3b3ff478fd8453e15cc22ab78fb898a5f90ec7f58bfa99affdcd4093fc888d04.webp)

Example:
Self Improving Coding Agents

### Anthropic’s July 2021 roadmap

[5:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=332s) [An excerpt describes a conversational AI assistant with the capabilities of a software engineering colleague.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f7115352aeb0f8c764c651512c7cf39b9899467d763b352cf6764cb4f84afd6.webp)

Anthropic’s roadmap in July 2021
Last Updated: July 2021
Long Term Goal
Our top level goal is to work towards the creation of safe, beneficial AGI.
As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc — it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it’s the direction in which we're striving.
Jared Kaplan, Concise Research Roadmap / Original PDF, page 2

### ChatGPT in 2022

[6:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=370s) [The slide dates the ChatGPT release to November 2022.](https://assets.ai.engineer/sites/aiecode2025/sha256/2150cbf2e8dece112d3db3391a57e14a0033066b1279ed5f7d8414bc965468c3.webp)

The 2022
- Chat GPT got released in Nov 2022

### Function calling in 2023–24

[6:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=379s) [A question asks what changes when a chat model can call functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/05f79ba5976a9239029a81a46bd3f5e301d20881fd29b9a6c0407a0242acee3d.webp)

The 2023-2024 intuition
- What is chats could call functions

### Sonnet 3.5 and agent tools

[6:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=395s) [The next slide connects the 2024 Sonnet 3.5 release with agents calling functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/448f93dccbc4ee04d9ecfa5ec9a77031d58d4f6870322006d7444d7b4da1cc87.webp)

2024 Sonnet 3.5 Release
What if agents could call functions

### Chatbots talking to each other

[6:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=415s) [The slide asks what would happen if chatbots could talk to each other.](https://assets.ai.engineer/sites/aiecode2025/sha256/43165ba114bf22de19bdfd4e8c6b2a61420b53ab34979b6e51625ef28b6683e8.webp)

The 2024 intuition
“What if the chatbots
talked to each other?”

### Multi-agent conversations

[7:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=422s) [An agent customization diagram shows joint and hierarchical conversation patterns and an example chat.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b9b36767c8d5d477ebcef88435fb7bf5522b6d58c6460915db6df99f691e995.webp)

Conversable agent
Agent Customization
Multi-Agent Conversations
Flexible Conversation Patterns
Joint chat
Hierarchical chat
Example Agent Chat
Output:

### Faster and cheaper agents

[7:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=454s) [Khan asks what happens if agents keep improving in speed and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7544aa42cb610473a64c7aa47bb22b67ac2f70a87083f6d8dd74ec62b2f7918.webp)

The 2025 intuition
What is agents just kept getting better
and a faster and cheaper

### An agent improving itself

[7:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=466s) [The slide asks whether an agent can improve itself.](https://assets.ai.engineer/sites/aiecode2025/sha256/02529e69a1b5b4ecd9eb798e435381ae41c8c7dcef9f8c10a3eb8a5f4402456d.webp)

The 2026 Intuition
What if the agent improved itself?

### Single-turn and agent evaluations

[7:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=478s) [Two diagrams contrast grading a model response with verifying the changes an agent makes to an environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/371326af90715e8f24599f8adb57e1011ca669de8d8affa96f7013ae2108a5d7.webp)

Comparison: Single-Turn vs Agent Evaluations
Single-turn
Prompt: How many toes does {ANIMAL} have?
Data: ANIMAL = Cat
LLM
Response
Grading logic: Correct if response == 18
Agent
Tools
Environment
Task: Set up an MCP server to connect to my app
Grading logic: Run a bunch of tests to make sure the MCP server is actually working

### Terminal-Bench

[9:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=555s) [The benchmark’s home page describes assessments for AI agents in terminal environments.](https://assets.ai.engineer/sites/aiecode2025/sha256/bfaf9569596d109ceb3312208b3d2c356e3c74979170979dbc03f567e2d92f02.webp)

terminal-bench: benchmarks for ai agents in terminal environments
terminal-bench is a collection of harbor-native benchmarks to help agent makers quantify their agents’ terminal mastery

### The evaluation harness

[9:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=587s) [A diagram separates tasks, graders, tracked metrics, traces and the agent harness.](https://assets.ai.engineer/sites/aiecode2025/sha256/76c38aaa2f9532f7101f8d8d564f51742fc797ad72c98f2168534259fbb48b59.webp)

Components of Evaluations for Agents
Evaluation harness
Evaluation suite
Task
Graders
Tracked metrics
Trials
Trace: trajectory, messages, tool calls, reasoning
Agent harness
Outcome
Grader evaluate

### Running 89 tasks in sandboxes

[10:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=618s) [The Harbor run diagram divides benchmark tasks among isolated sandboxes.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fc8d305f64d2616462b773d8d2fe6ccce59359190342d5548e4b404edf5ab34.webp)

User’s local machine
Harbor Dataset Download
Pull terminal bench dataset locally
List of Datasets: SWE Bench, SWE Bench pro, Terminal Bench, MMLU, Aider Polyglot
Harbor Run
Splits 89 sandboxes for 89 tasks

### From a score to a harness change

[10:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=643s) [A five-step process groups failures and changes one thing between evaluation runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/f33bf2e54a8dcfad27ac9d517c50b2ff8e8742e2f1defa22672573d56d2e5222.webp)

The Process: Get a Score, Then Improve
1. Run the eval: get an original score
2. Evaluate all failures: is there a common theme?
Example: a read_file tool blows up on massive files
Example: inference is so slow that individual runs time out
Example: the agent gets distracted and goes completely off the rails
3. Bucket your failures: 10 of this kind, 20 of that kind, 30 of another
4. Make one tiny change: re-run the whole eval — repeat
5. The agent iteratively figures out its flaws and improves

### Hill climbing on public evaluations

[11:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=702s) [The 2026 intuition slide introduces iterative improvement on public evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a95e17e678ec435c7faedc221fa585198ed9cf204fa05d2ac56282f07aad16a.webp)

The 2026 Intuition
Hill Climbing on Public Evals

### One agent improves another’s harness

[11:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=718s) [GPT-5.6 Sol led the experiment while Kimi K3 ran the tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4f3a6b654dabbb1ee891b52c7d96eb227af7d5e160cb04f45a553741cc56fde.webp)

Our Cline experiment
One agent improved the harness that another agent used
GPT-5.6 Sol led the work. Kimi K3 ran the tasks.

### The same 89 tasks

[12:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=777s) [The scorecard compares a baseline of 69/89 (77.5%) with confirmation at 79/89 (88.8%).](https://assets.ai.engineer/sites/aiecode2025/sha256/4842955759a16b687152e49d93b202150b24f355591a5d783d380e451666ac84.webp)

The same 89 tasks
69/89 · Baseline · 77.5%
79/89 · Confirmation · 88.8%

### The inner and outer learning loops

[13:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=827s) [The rollout agent acts on its environment while a separate meta-agent changes the harness and evaluates the result.](https://assets.ai.engineer/sites/aiecode2025/sha256/529eb69ad03efcca98894f44f458481b0a1a6dad54766f1271fe997540891167.webp)

Experience becomes a better harness.
A two-level learning loop, from individual tasks to reusable improvements.
INNER LOOP: Interact within each task
Rollout Agent · Harness · Model · Environment · feedback + reward
OUTER LOOP: Improve across tasks
Meta Agent · Diagnose & propose
Editable harness · Prompts · Tools · Skills · Context management
Seed tasks · Model
Evaluate & promote

### Model-assisted inference optimization

[14:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=889s) [OpenAI-reported figures on the slide distinguish 20% lower serving cost from more than 15% higher token-generation efficiency.](https://assets.ai.engineer/sites/aiecode2025/sha256/791c7f1f4da1a7c38319ac838bca4aa13d588952b1719d52e0abcf14039e4714.webp)

A model can improve its own inference
20% lower serving cost
>15% higher token-generation efficiency
OpenAI-reported results across distinct optimizations

### Rewriting production kernels

[14:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=899s) [A highlighted excerpt discusses kernel writing, kernel verification and end-to-end serving costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d7826960c3f4f29c2136d820fe1d8461a98afa57d08d4cab3f12e7113d31bc7.webp)

A model can improve its own inference
With Codex, GPT-5.6 Sol autonomously rewrote and optimized our production kernels, the core code that executes the mathematical operations that make up the model. This worked in part because we’ve trained GPT-5.6 to be effective at writing and improving kernels in Triton and Gluon, two open-source GPU programming languages maintained by OpenAI. These efforts, combined with broader kernel advancements from GPT-5.6 Sol, reduced end-to-end serving costs by 20%. We’ve also heavily invested in verification tooling, such as the open-source tool FpSan (Floating-Point Sanitizer), to help validate the correctness of the kernels written by GPT-5.6 Sol.

### An agent-assisted GPU competition entry

[15:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=940s) [A quoted competitor describes writing GPU operators with language models and agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/08a319dbf5bed19c7b054fbb769c7c48e4690593cb57e4f64ca3ffb080739b7d.webp)

A model can improve its own inference
Mark Saroufim
“I just learnt that one of the top submitters in the nvfp4 competition has never hand written GPU code before. I did not expect this.”
“That's me. Purely AI so far.”

### An improvement surface across the stack

[16:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=979s) [The slide names evaluation, harnesses, inference, training data, post-training and interaction with the world.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcb2c26ff5e654ac069ac979fb99f4dc7e5315270c418399a176e427a5c6f6ba.webp)

The improvement surface is the whole stack
Evaluation · Training data
Harness · Post-training
Inference · World interaction

### The whole recursive improvement loop

[16:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1011s) [A five-column diagram links pre-training, post-training, inference, eval generation and evals.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9d39c481d7c9aa04cd3c9ef6822480ba4593e5033fc56d22dff9796d85a5327.webp)

The whole RSI loop
AI can improve the machinery that builds and tests the next AI.
Pre-training: Nemotron-CC — AI-curated + rephrased data
Post-training: Constitutional AI — Self-critiques + AI preferences
Inference: GPT-5.6 Sol — Improves its own speculative decode
Eval generation: Bloom — Behavior description → new test suite
Evals: Self-Taught Evaluators — Synthetic preferences improve the judge
Validated gains feed better data, training, inference and tests into the next cycle.

### Human corrections as a training signal

[17:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1041s) [An attempt leads to human correction and then to a verified improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/c488239ed13e26dda3208a4dc0f1f83a5fb4ea75a7cb55650d0a2ebb678ed478.webp)

Human corrections become training signal
Attempt
Human correction
Verified improvement
A useful delta preserves what changed and why.

### Post-training closes another loop

[18:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1097s) [Experience, verified data, better experiments and a new checkpoint make up the proposed feedback cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/213392f6d7c05b32b51b8ff74492ae55224cd22bfaee0bee48c417f76b49b9b4.webp)

Post-training closes another loop
Experience · Verified data
Better experiments · New checkpoint
A proposed feedback loop, tested at each iteration

### Testing in the physical world

[18:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1133s) [A robot workspace illustrates a cycle of reset, act, verify and improve.](https://assets.ai.engineer/sites/aiecode2025/sha256/f23e4832046553daa3cce87afcdf6d6d2092efd5a2b80f53eb97b3424ffdcbe2.webp)

The physical world adds a harder test
Reset
Act
Verify
Improve

### Connecting the loops

[20:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1226s) [Cheaper inference buys more experiments per budget and creates opportunities for better data and models.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ab8dae4fc649b8b2e1325d02ad5061930d48af758f7d9466ecd10306f078da5.webp)

What happens when the loops connect?
Cheaper inference
More experiments per budget
Better data + models

### Which parts already work?

[21:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1298s) [Khan asks whether this is the singularity and observes that parts of the loop already exist.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff5c198dc72a678b2e430f7c469e63959440985b48a503e642a2ce9be031c9a3.webp)

Is this the singularity?
Parts of the loop are already real

### Move closer to the improvement loop

[21:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1313s) [Khan advises the audience to build where progress is verifiable, find bottlenecks and enable the next improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/14ee61211d8df8b7bb01b7daa5f7fdb565ec7dc70e300fd9da0683a1406e6505.webp)

Move closer to the RSI loop
Build where progress is verifiable.
Find the bottleneck.
Make the next improvement possible.
A career compass: pipelines, models, and measurable improvement.

### Choosing a project with a short time horizon

[23:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1438s) [The closing advice weighs ambitious collaborations, short project timescales and work that benefits from model scaling.](https://assets.ai.engineer/sites/aiecode2025/sha256/2025e0b956182e47316aff9b413c70a264eeba3b8f969469f1baec49bcc1f94c.webp)

I have 1–5 years to contribute intellectually to the world. What should I do?
Prefer ambitious, targeted, fast collaborations.
The timescale of the project needs to be shorter than the characteristic timescale of the exponential.
Otherwise, it's better to wait, and use a newer foundation model to do it better-faster.
This does not favor slow, independent, meandering exploration.
The bitter lesson [Sutton, 2019]
General methods that leverage computation are ultimately the most effective.
Work on projects that will benefit from scale.
Avoid projects that will be solved by scale alone, before you even finish the project.
2 years hard work

## Full transcript

[0:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=17s) Uh, hey guys, this is Ara. I'm a

[0:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=20s) Uh, hey guys, this is Ara. I'm a

[0:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=20s) Uh, hey guys, this is Ara. I'm a founding engineer at Cline and today I'm

[0:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=22s) founding engineer at Cline and today I'm

[0:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=22s) founding engineer at Cline and today I'm going to be talking about recursively

[0:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=24s) going to be talking about recursively

[0:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=24s) going to be talking about recursively self-improving coding agents. So, this

[0:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=26s) self-improving coding agents. So, this

[0:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=26s) self-improving coding agents. So, this is a very interesting conversation in

[0:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=30s) is a very interesting conversation in

[0:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=30s) is a very interesting conversation in the sense that everyone wants to

[0:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=32s) the sense that everyone wants to

[0:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=32s) the sense that everyone wants to understand what RSI is, how we

[0:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=34s) understand what RSI is, how we

[0:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=34s) understand what RSI is, how we accomplish it, how we go through it, and

[0:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=35s) accomplish it, how we go through it, and

[0:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=35s) accomplish it, how we go through it, and how it affects our career. So I'm going

[0:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=37s) how it affects our career. So I'm going

[0:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=37s) how it affects our career. So I'm going to start with giving you just three

[0:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=38s) to start with giving you just three

[0:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=38s) to start with giving you just three goals of this entire conversation. Just

[0:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=40s) goals of this entire conversation. Just

[0:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=40s) goals of this entire conversation. Just three simple goals and that would help

[0:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=43s) three simple goals and that would help

[0:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=43s) three simple goals and that would help you understand like what what we get out

[0:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=45s) you understand like what what we get out

[0:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=45s) you understand like what what we get out of it and what the future uh for this

[0:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=47s) of it and what the future uh for this

[0:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=47s) of it and what the future uh for this looks like. So the goal number first is

[0:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=49s) looks like. So the goal number first is

[0:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=49s) looks like. So the goal number first is that understanding the intent of

[0:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=50s) that understanding the intent of

[0:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=50s) that understanding the intent of frontier labs right everyone who's

[0:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=52s) frontier labs right everyone who's

[0:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=52s) frontier labs right everyone who's outside of the frontier labs is always

[0:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=53s) outside of the frontier labs is always

[0:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=53s) outside of the frontier labs is always trying to speculate like hey what what

[0:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=55s) trying to speculate like hey what what

[0:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=55s) trying to speculate like hey what what is the topic in open or what is deep

[0:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=57s) is the topic in open or what is deep

[0:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=57s) is the topic in open or what is deep mind going to come out with or what what

[0:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=58s) mind going to come out with or what what

[0:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=58s) mind going to come out with or what what are they trying to make and what's the

[1:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=60s) are they trying to make and what's the

[1:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=60s) are they trying to make and what's the future going to look like who has

[1:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=61s) future going to look like who has

[1:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=62s) future going to look like who has cracked the RSI. So like this kind of

[1:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=64s) cracked the RSI. So like this kind of

[1:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=64s) cracked the RSI. So like this kind of helps you get get a rough outline of

[1:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=66s) helps you get get a rough outline of

[1:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=66s) helps you get get a rough outline of what their road map looks like. And the

[1:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=68s) what their road map looks like. And the

[1:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=68s) what their road map looks like. And the second thing is that like if the models

[1:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=70s) second thing is that like if the models

[1:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=70s) second thing is that like if the models are improving so fast like how do you

[1:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=72s) are improving so fast like how do you

[1:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=72s) are improving so fast like how do you think about your career or the next few

[1:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=74s) think about your career or the next few

[1:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=74s) think about your career or the next few years right that is really what we want

[1:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=76s) years right that is really what we want

[1:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=76s) years right that is really what we want to understand. And we want to understand

[1:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=77s) to understand. And we want to understand

[1:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=77s) to understand. And we want to understand like okay things are changing so fast

[1:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=80s) like okay things are changing so fast

[1:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=80s) like okay things are changing so fast like the ground underneath us is

[1:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=82s) like the ground underneath us is

[1:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=82s) like the ground underneath us is shifting so quickly right now like it's

[1:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=84s) shifting so quickly right now like it's

[1:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=84s) shifting so quickly right now like it's like uh

[1:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=86s) like uh

[1:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=86s) like uh there used to be a time when you could

[1:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=88s) there used to be a time when you could

[1:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=88s) there used to be a time when you could like demonstrate your side projects on

[1:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=91s) like demonstrate your side projects on

[1:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=91s) like demonstrate your side projects on your personal website or GitHub and now

[1:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=93s) your personal website or GitHub and now

[1:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=93s) your personal website or GitHub and now it's just like you could literally

[1:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=94s) it's just like you could literally

[1:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=94s) it's just like you could literally oneshot them using any Frontier coding

[1:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=97s) oneshot them using any Frontier coding

[1:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=97s) oneshot them using any Frontier coding engine. And with that it's like it's

[1:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=99s) engine. And with that it's like it's

[1:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=99s) engine. And with that it's like it's like oh how how do we demonstrate proof

[1:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=101s) like oh how how do we demonstrate proof

[1:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=101s) like oh how how do we demonstrate proof of work? How do you demonstrate, you

[1:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=103s) of work? How do you demonstrate, you

[1:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=103s) of work? How do you demonstrate, you know, that that that we know what we're

[1:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=105s) know, that that that we know what we're

[1:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=105s) know, that that that we know what we're doing and how do we think of our career

[1:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=107s) doing and how do we think of our career

[1:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=107s) doing and how do we think of our career for the next coming years, right? So,

[1:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=109s) for the next coming years, right? So,

[1:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=109s) for the next coming years, right? So, it's a very interesting times that we're

[1:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=111s) it's a very interesting times that we're

[1:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=111s) it's a very interesting times that we're living in. And then the third and the

[1:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=112s) living in. And then the third and the

[1:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=112s) living in. And then the third and the most important goal of the conversation

[1:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=114s) most important goal of the conversation

[1:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=114s) most important goal of the conversation is escaping the permanent underclass,

[1:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=116s) is escaping the permanent underclass,

[1:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=116s) is escaping the permanent underclass, right? So, I know this seems like a

[1:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=118s) right? So, I know this seems like a

[1:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=118s) right? So, I know this seems like a joke, but this really is a very valid

[2:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=121s) joke, but this really is a very valid

[2:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=121s) joke, but this really is a very valid possibility that people are

[2:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=122s) possibility that people are

[2:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=122s) possibility that people are contemplating. Everyone's like, "Oh,

[2:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=124s) contemplating. Everyone's like, "Oh,

[2:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=124s) contemplating. Everyone's like, "Oh, well, if everything that we want to be

[2:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=126s) well, if everything that we want to be

[2:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=126s) well, if everything that we want to be doing can potentially easily be done by

[2:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=128s) doing can potentially easily be done by

[2:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=128s) doing can potentially easily be done by a frontier model, like what are we what

[2:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=129s) a frontier model, like what are we what

[2:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=129s) a frontier model, like what are we what are we good for?" And like where does

[2:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=131s) are we good for?" And like where does

[2:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=131s) are we good for?" And like where does software go? Where where where does the

[2:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=133s) software go? Where where where does the

[2:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=133s) software go? Where where where does the future of technology going to go? And

[2:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=134s) future of technology going to go? And

[2:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=134s) future of technology going to go? And what are we going to do as as things

[2:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=136s) what are we going to do as as things

[2:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=136s) what are we going to do as as things evolve so fast, right? So I think that

[2:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=139s) evolve so fast, right? So I think that

[2:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=139s) evolve so fast, right? So I think that there's like a lot of like

[2:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=140s) there's like a lot of like

[2:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=140s) there's like a lot of like fear-mongering but also there's like

[2:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=142s) fear-mongering but also there's like

[2:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=142s) fear-mongering but also there's like some very valid merge where people are

[2:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=145s) some very valid merge where people are

[2:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=145s) some very valid merge where people are concerned that like if all the revenue

[2:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=147s) concerned that like if all the revenue

[2:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=147s) concerned that like if all the revenue of the planet is going to go to um a few

[2:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=149s) of the planet is going to go to um a few

[2:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=150s) of the planet is going to go to um a few hardware companies that make frontier

[2:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=151s) hardware companies that make frontier

[2:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=151s) hardware companies that make frontier GPUs and then a few few uh AI companies

[2:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=155s) GPUs and then a few few uh AI companies

[2:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=155s) GPUs and then a few few uh AI companies that that make the best models like how

[2:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=157s) that that make the best models like how

[2:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=158s) that that make the best models like how are we how are the rest of us going to

[2:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=159s) are we how are the rest of us going to

[2:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=159s) are we how are the rest of us going to thrive and what what are we going to do

[2:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=161s) thrive and what what are we going to do

[2:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=161s) thrive and what what are we going to do here? So, I think I'll I'll walk you

[2:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=163s) here? So, I think I'll I'll walk you

[2:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=163s) here? So, I think I'll I'll walk you through a few examples and a few ideas

[2:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=165s) through a few examples and a few ideas

[2:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=165s) through a few examples and a few ideas that will like really really help you

[2:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=167s) that will like really really help you

[2:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=167s) that will like really really help you reconsider the career choices and and

[2:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=169s) reconsider the career choices and and

[2:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=169s) reconsider the career choices and and and the life choices you're making and

[2:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=171s) and the life choices you're making and

[2:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=171s) and the life choices you're making and what your problems you're picking up and

[2:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=173s) what your problems you're picking up and

[2:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=173s) what your problems you're picking up and then and then that will help you like

[2:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=175s) then and then that will help you like

[2:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=175s) then and then that will help you like realign

[2:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=176s) realign

[2:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=176s) realign what it is that we should be working on

[2:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=178s) what it is that we should be working on

[2:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=178s) what it is that we should be working on and what it is that you you know would

[3:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=180s) and what it is that you you know would

[3:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=180s) and what it is that you you know would would that you could work on that will

[3:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=182s) would that you could work on that will

[3:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=182s) would that you could work on that will stand the test of time and help you both

[3:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=185s) stand the test of time and help you both

[3:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=185s) stand the test of time and help you both just not just like on a psychological

[3:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=186s) just not just like on a psychological

[3:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=186s) just not just like on a psychological but like financial and other levels as

[3:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=188s) but like financial and other levels as

[3:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=188s) but like financial and other levels as well. So uh to give you a very precise

[3:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=190s) well. So uh to give you a very precise

[3:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=190s) well. So uh to give you a very precise example, let's start with something very

[3:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=191s) example, let's start with something very

[3:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=191s) example, let's start with something very precise that I had done. So a couple

[3:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=194s) precise that I had done. So a couple

[3:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=194s) precise that I had done. So a couple months ago I uh came up with recursive

[3:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=198s) months ago I uh came up with recursive

[3:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=198s) months ago I uh came up with recursive shampoo improvements on a coding

[3:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=201s) shampoo improvements on a coding

[3:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=201s) shampoo improvements on a coding harness. So this was the first public

[3:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=203s) harness. So this was the first public

[3:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=203s) harness. So this was the first public example of uh an automatic eval

[3:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=206s) example of uh an automatic eval

[3:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=206s) example of uh an automatic eval improvement. So to give you an example

[3:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=208s) improvement. So to give you an example

[3:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=208s) improvement. So to give you an example um we used to do this thing where we

[3:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=210s) um we used to do this thing where we

[3:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=210s) um we used to do this thing where we would measure the quality of a coding

[3:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=212s) would measure the quality of a coding

[3:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=212s) would measure the quality of a coding agent using evaluation scores. So on

[3:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=215s) agent using evaluation scores. So on

[3:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=215s) agent using evaluation scores. So on Terminal-Bench 2.1 Kimi K3 with client

[3:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=218s) Terminal-Bench 2.1 Kimi K3 with client

[3:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=218s) Terminal-Bench 2.1 Kimi K3 with client would score um usually like you know um

[3:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=222s) would score um usually like you know um

[3:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=222s) would score um usually like you know um 60 70% and then depending on like how

[3:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=225s) 60 70% and then depending on like how

[3:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=225s) 60 70% and then depending on like how good the model is like you get better

[3:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=226s) good the model is like you get better

[3:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=226s) good the model is like you get better scores on on Terminal-Bench 2.1. So we

[3:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=229s) scores on on Terminal-Bench 2.1. So we

[3:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=229s) scores on on Terminal-Bench 2.1. So we noticed that when Kimi K3 came out, our

[3:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=231s) noticed that when Kimi K3 came out, our

[3:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=231s) noticed that when Kimi K3 came out, our score was 77.5%. Which is a good score,

[3:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=233s) score was 77.5%. Which is a good score,

[3:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=233s) score was 77.5%. Which is a good score, but we were like, "Oh, is this is this

[3:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=235s) but we were like, "Oh, is this is this

[3:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=235s) but we were like, "Oh, is this is this truly is it truly the best score that

[3:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=237s) truly is it truly the best score that

[3:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=237s) truly is it truly the best score that this model can get on this benchmark?

[3:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=239s) this model can get on this benchmark?

[4:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=240s) this model can get on this benchmark? And also like what does the model

[4:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=242s) And also like what does the model

[4:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=242s) And also like what does the model release paper claim about what their

[4:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=244s) release paper claim about what their

[4:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=244s) release paper claim about what their best score is?" So the model's release

[4:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=245s) best score is?" So the model's release

[4:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=245s) best score is?" So the model's release paper said that they could score up to

[4:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=247s) paper said that they could score up to

[4:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=247s) paper said that they could score up to 85% or even more. So we were just like,

[4:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=249s) 85% or even more. So we were just like,

[4:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=249s) 85% or even more. So we were just like, okay, okay, so we are not getting the

[4:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=251s) okay, okay, so we are not getting the

[4:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=251s) okay, okay, so we are not getting the sort of score. And often what we do with

[4:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=253s) sort of score. And often what we do with

[4:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=254s) sort of score. And often what we do with model releases in client is that we try

[4:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=255s) model releases in client is that we try

[4:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=255s) model releases in client is that we try to like optimize your score and try to

[4:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=257s) to like optimize your score and try to

[4:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=257s) to like optimize your score and try to get the best out of the model. So when

[4:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=260s) get the best out of the model. So when

[4:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=260s) get the best out of the model. So when we started hill climbing on the on the

[4:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=262s) we started hill climbing on the on the

[4:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=262s) we started hill climbing on the on the model like we we realized this we

[4:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=265s) model like we we realized this we

[4:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=265s) model like we we realized this we realized that like you know there's a

[4:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=267s) realized that like you know there's a

[4:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=267s) realized that like you know there's a lot of steps that we do says that we do

[4:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=268s) lot of steps that we do says that we do

[4:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=268s) lot of steps that we do says that we do it gracefully says that we pursue the

[4:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=270s) it gracefully says that we pursue the

[4:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=270s) it gracefully says that we pursue the right kind of problems and do it the

[4:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=271s) right kind of problems and do it the

[4:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=271s) right kind of problems and do it the right way. And we started thinking like

[4:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=275s) right way. And we started thinking like

[4:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=275s) right way. And we started thinking like okay if we there's a lot of ground work

[4:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=277s) okay if we there's a lot of ground work

[4:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=277s) okay if we there's a lot of ground work here how do we automate it and this is a

[4:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=279s) here how do we automate it and this is a

[4:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=279s) here how do we automate it and this is a demonstration of like how we automated

[4:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=280s) demonstration of like how we automated

[4:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=280s) demonstration of like how we automated it. So I'll I'll just leave that as a

[4:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=282s) it. So I'll I'll just leave that as a

[4:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=282s) it. So I'll I'll just leave that as a placeholder for now and we'll get get

[4:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=283s) placeholder for now and we'll get get

[4:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=283s) placeholder for now and we'll get get back to it later where we talk about in

[4:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=285s) back to it later where we talk about in

[4:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=285s) back to it later where we talk about in detail of like how does like this

[4:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=287s) detail of like how does like this

[4:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=287s) detail of like how does like this automatic improvement loop of a client

[4:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=289s) automatic improvement loop of a client

[4:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=289s) automatic improvement loop of a client harness work. So this basically means

[4:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=291s) harness work. So this basically means

[4:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=292s) harness work. So this basically means that like you have a coding agent

[4:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=293s) that like you have a coding agent

[4:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=293s) that like you have a coding agent harness that could just like basically

[4:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=294s) harness that could just like basically

[4:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=294s) harness that could just like basically improve itself and become better on on

[4:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=296s) improve itself and become better on on

[4:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=296s) improve itself and become better on on the same set of problems. So let's start

[4:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=299s) the same set of problems. So let's start

[4:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=299s) the same set of problems. So let's start with the ground terms. What is recursive

[5:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=300s) with the ground terms. What is recursive

[5:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=300s) with the ground terms. What is recursive self improvement? Recursive

[5:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=301s) self improvement? Recursive

[5:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=301s) self improvement? Recursive self-improvement is the idea that an AI

[5:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=303s) self-improvement is the idea that an AI

[5:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=303s) self-improvement is the idea that an AI system that helps [snorts] build a more

[5:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=306s) system that helps [snorts] build a more

[5:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=306s) system that helps [snorts] build a more capable successor which repeats the

[5:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=308s) capable successor which repeats the

[5:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=308s) capable successor which repeats the process you know until you hit

[5:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=310s) process you know until you hit

[5:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=310s) process you know until you hit artificial general intelligence right

[5:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=311s) artificial general intelligence right

[5:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=311s) artificial general intelligence right you have better systems you have better

[5:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=312s) you have better systems you have better

[5:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=312s) you have better systems you have better improvements you have a better system

[5:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=314s) improvements you have a better system

[5:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=314s) improvements you have a better system and then you know you have singularity

[5:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=316s) and then you know you have singularity

[5:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=316s) and then you know you have singularity so a classic example is like a

[5:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=317s) so a classic example is like a

[5:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=318s) so a classic example is like a self-reing coding agent right

[5:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=319s) self-reing coding agent right

[5:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=319s) self-reing coding agent right self-reving coding agent is just like a

[5:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=321s) self-reving coding agent is just like a

[5:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=321s) self-reving coding agent is just like a coding agent that can improve itself so

[5:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=323s) coding agent that can improve itself so

[5:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=323s) coding agent that can improve itself so you know this this is all like a pie in

[5:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=325s) you know this this is all like a pie in

[5:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=325s) you know this this is all like a pie in the sky idea like it's like it's like

[5:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=327s) the sky idea like it's like it's like

[5:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=327s) the sky idea like it's like it's like you know yeah sure this looks cool but

[5:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=328s) you know yeah sure this looks cool but

[5:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=328s) you know yeah sure this looks cool but how how did this even come out to be

[5:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=330s) how how did this even come out to be

[5:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=330s) how how did this even come out to be like how did we even get to here? So if

[5:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=333s) like how did we even get to here? So if

[5:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=333s) like how did we even get to here? So if you notice in topics road map in 2021 in

[5:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=336s) you notice in topics road map in 2021 in

[5:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=336s) you notice in topics road map in 2021 in 2021's road map they very specifically

[5:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=338s) 2021's road map they very specifically

[5:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=338s) 2021's road map they very specifically said that uh uh they wanted to develop a

[5:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=342s) said that uh uh they wanted to develop a

[5:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=342s) said that uh uh they wanted to develop a general AI assistant that can interact

[5:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=343s) general AI assistant that can interact

[5:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=343s) general AI assistant that can interact conversationally with a user to create

[5:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=346s) conversationally with a user to create

[5:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=346s) conversationally with a user to create edit and explain software. It would

[5:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=347s) edit and explain software. It would

[5:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=348s) edit and explain software. It would display all the capabilities of a

[5:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=349s) display all the capabilities of a

[5:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=349s) display all the capabilities of a software engineer colleague. It would

[5:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=350s) software engineer colleague. It would

[5:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=350s) software engineer colleague. It would write functions. It would identify bugs.

[5:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=352s) write functions. It would identify bugs.

[5:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=352s) write functions. It would identify bugs. It would design an API discuss style and

[5:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=354s) It would design an API discuss style and

[5:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=354s) It would design an API discuss style and best practices. Right? So we don't

[5:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=356s) best practices. Right? So we don't

[5:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=356s) best practices. Right? So we don't expect to reach this goal right away but

[5:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=358s) expect to reach this goal right away but

[5:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=358s) expect to reach this goal right away but it's in the direction which we're

[5:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=359s) it's in the direction which we're

[5:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=359s) it's in the direction which we're striving. It's interesting how this was

[6:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=361s) striving. It's interesting how this was

[6:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=361s) striving. It's interesting how this was more than five years ago. More than five

[6:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=363s) more than five years ago. More than five

[6:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=363s) more than five years ago. More than five years ago, people had predicted that

[6:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=365s) years ago, people had predicted that

[6:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=365s) years ago, people had predicted that there exist a day where you will have

[6:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=367s) there exist a day where you will have

[6:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=367s) there exist a day where you will have like a full-blown journal assistant that

[6:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=369s) like a full-blown journal assistant that

[6:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=369s) like a full-blown journal assistant that can basically act like a colleague,

[6:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=370s) can basically act like a colleague,

[6:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=370s) can basically act like a colleague, which we do have right now. Um, then in

[6:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=373s) which we do have right now. Um, then in

[6:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=373s) which we do have right now. Um, then in 2022, Chad Deputy got released. We all

[6:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=375s) 2022, Chad Deputy got released. We all

[6:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=375s) 2022, Chad Deputy got released. We all know how that happened. It was it was a

[6:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=377s) know how that happened. It was it was a

[6:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=377s) know how that happened. It was it was a fun moment. It was like the first time

[6:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=379s) fun moment. It was like the first time

[6:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=379s) fun moment. It was like the first time things got sparked for us. Um, then in

[6:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=382s) things got sparked for us. Um, then in

[6:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=382s) things got sparked for us. Um, then in 2023, 2024, there was the first time

[6:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=384s) 2023, 2024, there was the first time

[6:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=384s) 2023, 2024, there was the first time chat models could call functions. These

[6:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=387s) chat models could call functions. These

[6:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=387s) chat models could call functions. These functions would be simple functions like

[6:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=389s) functions would be simple functions like

[6:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=389s) functions would be simple functions like hey what's the weather or you know

[6:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=391s) hey what's the weather or you know

[6:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=391s) hey what's the weather or you know what's the what's you know what's this

[6:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=393s) what's the what's you know what's this

[6:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=393s) what's the what's you know what's this computation or what this computations

[6:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=395s) computation or what this computations

[6:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=395s) computation or what this computations result in python um and then finally in

[6:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=398s) result in python um and then finally in

[6:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=398s) result in python um and then finally in 2024 when s 345 was really easy that was

[6:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=400s) 2024 when s 345 was really easy that was

[6:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=400s) 2024 when s 345 was really easy that was when the agents could like truly call

[6:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=402s) when the agents could like truly call

[6:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=402s) when the agents could like truly call like actual tools the way we call like

[6:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=405s) like actual tools the way we call like

[6:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=405s) like actual tools the way we call like think of tools so it's like reading a

[6:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=407s) think of tools so it's like reading a

[6:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=407s) think of tools so it's like reading a file editing a file you know reading

[6:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=409s) file editing a file you know reading

[6:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=409s) file editing a file you know reading searching something on web uh listing

[6:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=411s) searching something on web uh listing

[6:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=411s) searching something on web uh listing listing a folder searching and grapping

[6:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=413s) listing a folder searching and grapping

[6:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=413s) listing a folder searching and grapping through you know uh a file system things

[6:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=415s) through you know uh a file system things

[6:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=415s) through you know uh a file system things like that. So, and in 2024, we had the

[6:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=419s) like that. So, and in 2024, we had the

[6:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=419s) like that. So, and in 2024, we had the intuition that like chatbot, what if

[7:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=421s) intuition that like chatbot, what if

[7:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=421s) intuition that like chatbot, what if chatbots could actually talk to each

[7:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=422s) chatbots could actually talk to each

[7:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=422s) chatbots could actually talk to each other, you know? So, then that's was the

[7:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=424s) other, you know? So, then that's was the

[7:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=424s) other, you know? So, then that's was the first time when we had like, you know,

[7:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=426s) first time when we had like, you know,

[7:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=426s) first time when we had like, you know, uh multi-agent conversations. Now, at

[7:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=428s) uh multi-agent conversations. Now, at

[7:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=428s) uh multi-agent conversations. Now, at the time, these things were very janky,

[7:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=429s) the time, these things were very janky,

[7:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=430s) the time, these things were very janky, right? like they were just like taped

[7:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=431s) right? like they were just like taped

[7:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=431s) right? like they were just like taped together in this like weird weird uh

[7:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=434s) together in this like weird weird uh

[7:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=434s) together in this like weird weird uh bizarre ways where you have just like

[7:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=436s) bizarre ways where you have just like

[7:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=436s) bizarre ways where you have just like these classes and these links and then

[7:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=438s) these classes and these links and then

[7:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=438s) these classes and these links and then like these hierarchies of classes and it

[7:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=440s) like these hierarchies of classes and it

[7:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=440s) like these hierarchies of classes and it was like it was all like kind of like a

[7:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=442s) was like it was all like kind of like a

[7:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=442s) was like it was all like kind of like a very janky attempt to like make us think

[7:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=444s) very janky attempt to like make us think

[7:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=444s) very janky attempt to like make us think that like these these models could like

[7:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=446s) that like these these models could like

[7:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=446s) that like these these models could like have these pipeline of flows of

[7:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=447s) have these pipeline of flows of

[7:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=447s) have these pipeline of flows of responses that can all communicate with

[7:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=449s) responses that can all communicate with

[7:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=449s) responses that can all communicate with each other but a [snorts] lot of it was

[7:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=451s) each other but a [snorts] lot of it was

[7:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=451s) each other but a [snorts] lot of it was like mostly deterministic and it was

[7:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=453s) like mostly deterministic and it was

[7:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=453s) like mostly deterministic and it was like you know it was not as free flowing

[7:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=454s) like you know it was not as free flowing

[7:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=454s) like you know it was not as free flowing as it is right now. So, and you know, we

[7:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=458s) as it is right now. So, and you know, we

[7:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=458s) as it is right now. So, and you know, we started getting the intuition. So, this

[7:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=460s) started getting the intuition. So, this

[7:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=460s) started getting the intuition. So, this was the autogen project, but like in

[7:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=461s) was the autogen project, but like in

[7:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=461s) was the autogen project, but like in only in 2025, we we [snorts] started

[7:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=463s) only in 2025, we we [snorts] started

[7:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=463s) only in 2025, we we [snorts] started getting the like what if the agents just

[7:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=465s) getting the like what if the agents just

[7:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=465s) getting the like what if the agents just kept getting better and faster and

[7:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=466s) kept getting better and faster and

[7:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=466s) kept getting better and faster and cheaper, right? What if that just kept

[7:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=468s) cheaper, right? What if that just kept

[7:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=468s) cheaper, right? What if that just kept happening? And then in 2026, uh we we

[7:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=471s) happening? And then in 2026, uh we we

[7:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=471s) happening? And then in 2026, uh we we had the intuition that like what if what

[7:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=474s) had the intuition that like what if what

[7:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=474s) had the intuition that like what if what if the agent could improve itself, which

[7:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=475s) if the agent could improve itself, which

[7:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=476s) if the agent could improve itself, which is like the final, you know, the final

[7:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=478s) is like the final, you know, the final

[7:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=478s) is like the final, you know, the final part of the whole process. So, let me

[8:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=480s) part of the whole process. So, let me

[8:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=480s) part of the whole process. So, let me give an example like what does it mean

[8:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=481s) give an example like what does it mean

[8:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=481s) give an example like what does it mean for an agent to improve itself, right?

[8:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=483s) for an agent to improve itself, right?

[8:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=483s) for an agent to improve itself, right? So like it's like a lot of times when we

[8:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=485s) So like it's like a lot of times when we

[8:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=485s) So like it's like a lot of times when we think of like evaluating an agent, we

[8:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=487s) think of like evaluating an agent, we

[8:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=487s) think of like evaluating an agent, we initially used to think of evaluating an

[8:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=489s) initially used to think of evaluating an

[8:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=489s) initially used to think of evaluating an agent in the form of like just a single

[8:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=491s) agent in the form of like just a single

[8:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=491s) agent in the form of like just a single turn conversation. So we would tell we

[8:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=493s) turn conversation. So we would tell we

[8:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=493s) turn conversation. So we would tell we would tell like um um agents that like

[8:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=496s) would tell like um um agents that like

[8:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=496s) would tell like um um agents that like ask a simple question like how many toes

[8:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=499s) ask a simple question like how many toes

[8:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=499s) ask a simple question like how many toes does a dash animal has like it could be

[8:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=501s) does a dash animal has like it could be

[8:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=501s) does a dash animal has like it could be let's say how many toes does a cat have

[8:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=503s) let's say how many toes does a cat have

[8:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=503s) let's say how many toes does a cat have and then al would go through and the

[8:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=504s) and then al would go through and the

[8:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=504s) and then al would go through and the element would like prepare response and

[8:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=506s) element would like prepare response and

[8:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=506s) element would like prepare response and then we would get the response. So like

[8:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=507s) then we would get the response. So like

[8:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=507s) then we would get the response. So like a cat has maybe 10 toes or 18 toes,

[8:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=510s) a cat has maybe 10 toes or 18 toes,

[8:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=510s) a cat has maybe 10 toes or 18 toes, however many. I don't I actually don't

[8:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=511s) however many. I don't I actually don't

[8:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=511s) however many. I don't I actually don't know how many toes the cat has. But

[8:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=513s) know how many toes the cat has. But

[8:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=513s) know how many toes the cat has. But anyway, um yeah. So then we create the

[8:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=517s) anyway, um yeah. So then we create the

[8:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=517s) anyway, um yeah. So then we create the response as like a yes or no. But like

[8:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=518s) response as like a yes or no. But like

[8:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=518s) response as like a yes or no. But like when you work with an agent, there's a

[8:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=520s) when you work with an agent, there's a

[8:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=520s) when you work with an agent, there's a ton of things that you're doing. So let

[8:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=522s) ton of things that you're doing. So let

[8:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=522s) ton of things that you're doing. So let me give an example. Let me give an

[8:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=523s) me give an example. Let me give an

[8:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=523s) me give an example. Let me give an example. So for example, you're going to

[8:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=525s) example. So for example, you're going to

[8:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=525s) example. So for example, you're going to ask an agent like hey set up this MCP

[8:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=527s) ask an agent like hey set up this MCP

[8:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=527s) ask an agent like hey set up this MCP for me. So now it's not just like it's

[8:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=529s) for me. So now it's not just like it's

[8:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=529s) for me. So now it's not just like it's not just like you know a trivial oneable

[8:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=532s) not just like you know a trivial oneable

[8:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=532s) not just like you know a trivial oneable like thing like it will like search

[8:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=533s) like thing like it will like search

[8:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=533s) like thing like it will like search online and go oh what's the MCP? what is

[8:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=536s) online and go oh what's the MCP? what is

[8:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=536s) online and go oh what's the MCP? what is the config? So, I'll like edit and

[8:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=538s) the config? So, I'll like edit and

[8:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=538s) the config? So, I'll like edit and download some soft files. Like maybe I

[8:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=539s) download some soft files. Like maybe I

[9:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=540s) download some soft files. Like maybe I need to add database update the MCB

[9:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=542s) need to add database update the MCB

[9:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=542s) need to add database update the MCB tool. Maybe I need to read and edit some

[9:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=544s) tool. Maybe I need to read and edit some

[9:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=544s) tool. Maybe I need to read and edit some files. Maybe I need to run some bash

[9:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=545s) files. Maybe I need to run some bash

[9:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=546s) files. Maybe I need to run some bash commands. Right? So, you do all of that

[9:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=547s) commands. Right? So, you do all of that

[9:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=548s) commands. Right? So, you do all of that and then the agent response like, hey, I

[9:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=549s) and then the agent response like, hey, I

[9:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=549s) and then the agent response like, hey, I did it. And then you grab that with a

[9:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=551s) did it. And then you grab that with a

[9:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=551s) did it. And then you grab that with a bunch of tests to gauge like, hey, did

[9:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=553s) bunch of tests to gauge like, hey, did

[9:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=553s) bunch of tests to gauge like, hey, did the agent actually accomplish the thing

[9:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=555s) the agent actually accomplish the thing

[9:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=555s) the agent actually accomplish the thing that it said it would do. So to do that

[9:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=558s) that it said it would do. So to do that

[9:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=558s) that it said it would do. So to do that thing the second part which is like more

[9:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=560s) thing the second part which is like more

[9:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=560s) thing the second part which is like more intricate where you're actually like

[9:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=562s) intricate where you're actually like

[9:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=562s) intricate where you're actually like truly you know testing agents like uh to

[9:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=565s) truly you know testing agents like uh to

[9:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=565s) truly you know testing agents like uh to their best extent like it's like you

[9:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=567s) their best extent like it's like you

[9:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=567s) their best extent like it's like you know um we use this thing called

[9:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=569s) know um we use this thing called

[9:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=569s) know um we use this thing called Terminal-Bench Terminal-Bench is a

[9:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=571s) Terminal-Bench Terminal-Bench is a

[9:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=571s) Terminal-Bench Terminal-Bench is a benchmark for you know AI agents in

[9:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=573s) benchmark for you know AI agents in

[9:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=573s) benchmark for you know AI agents in terminal environments and that is

[9:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=574s) terminal environments and that is

[9:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=574s) terminal environments and that is something that we we have a lot of like

[9:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=577s) something that we we have a lot of like

[9:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=577s) something that we we have a lot of like experience with at client and uh it's

[9:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=579s) experience with at client and uh it's

[9:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=579s) experience with at client and uh it's like a it's like one of the best ways to

[9:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=581s) like a it's like one of the best ways to

[9:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=581s) like a it's like one of the best ways to like really gauge like is your coding

[9:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=582s) like really gauge like is your coding

[9:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=582s) like really gauge like is your coding agent accomplishing uh in general the

[9:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=585s) agent accomplishing uh in general the

[9:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=585s) agent accomplishing uh in general the task that you're doing uh that you want

[9:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=587s) task that you're doing uh that you want

[9:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=587s) task that you're doing uh that you want to be doing with a hudded model. So

[9:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=589s) to be doing with a hudded model. So

[9:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=589s) to be doing with a hudded model. So there's a couple there's a many

[9:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=590s) there's a couple there's a many

[9:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=590s) there's a couple there's a many different parts of a validation suit. So

[9:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=593s) different parts of a validation suit. So

[9:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=593s) different parts of a validation suit. So ideally you want that to run in um

[9:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=596s) ideally you want that to run in um

[9:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=596s) ideally you want that to run in um [snorts] you know you you want that to

[9:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=598s) [snorts] you know you you want that to

[9:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=598s) [snorts] you know you you want that to run in um individual sandboxes. Then you

[10:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=601s) run in um individual sandboxes. Then you

[10:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=601s) run in um individual sandboxes. Then you have like independent graders that like

[10:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=603s) have like independent graders that like

[10:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=603s) have like independent graders that like check everything like how many turns it

[10:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=605s) check everything like how many turns it

[10:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=605s) check everything like how many turns it took, how many tokens it took, how much

[10:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=606s) took, how many tokens it took, how much

[10:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=606s) took, how many tokens it took, how much latency it took, how many how many how

[10:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=608s) latency it took, how many how many how

[10:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=608s) latency it took, how many how many how many tool calls did it make, what was

[10:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=610s) many tool calls did it make, what was

[10:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=610s) many tool calls did it make, what was the trajectory looking like and then

[10:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=611s) the trajectory looking like and then

[10:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=612s) the trajectory looking like and then grading all the results for things like

[10:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=613s) grading all the results for things like

[10:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=613s) grading all the results for things like hey did you not reward hack? Did you not

[10:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=615s) hey did you not reward hack? Did you not

[10:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=615s) hey did you not reward hack? Did you not cheat through the solution? do not just

[10:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=616s) cheat through the solution? do not just

[10:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=616s) cheat through the solution? do not just like look up the solution online things

[10:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=618s) like look up the solution online things

[10:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=618s) like look up the solution online things like that. Um so we we basically like a

[10:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=622s) like that. Um so we we basically like a

[10:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=622s) like that. Um so we we basically like a lot of these uh data sets of evaluation

[10:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=624s) lot of these uh data sets of evaluation

[10:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=624s) lot of these uh data sets of evaluation problems have a set of tasks. So when

[10:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=627s) problems have a set of tasks. So when

[10:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=627s) problems have a set of tasks. So when here's a Terminal-Bench 2.1 they have 89

[10:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=629s) here's a Terminal-Bench 2.1 they have 89

[10:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=629s) here's a Terminal-Bench 2.1 they have 89 tasks. So what we do is like we spin up

[10:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=631s) tasks. So what we do is like we spin up

[10:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=631s) tasks. So what we do is like we spin up 89 parallel containers where they all

[10:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=634s) 89 parallel containers where they all

[10:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=634s) 89 parallel containers where they all run online and they run in parallel and

[10:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=636s) run online and they run in parallel and

[10:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=636s) run online and they run in parallel and they're sandbox and then they each try

[10:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=639s) they're sandbox and then they each try

[10:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=639s) they're sandbox and then they each try to solve a problem. So then they're all

[10:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=641s) to solve a problem. So then they're all

[10:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=641s) to solve a problem. So then they're all uh in parallel and independent of each

[10:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=643s) uh in parallel and independent of each

[10:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=643s) uh in parallel and independent of each other and then at the end we get get a

[10:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=645s) other and then at the end we get get a

[10:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=645s) other and then at the end we get get a score. So our score could like give us

[10:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=647s) score. So our score could like give us

[10:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=647s) score. So our score could like give us measures like is is a is this is the

[10:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=650s) measures like is is a is this is the

[10:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=650s) measures like is is a is this is the model performing well and what kind of

[10:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=652s) model performing well and what kind of

[10:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=652s) model performing well and what kind of failures it has right? So like you run

[10:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=654s) failures it has right? So like you run

[10:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=654s) failures it has right? So like you run the evaluation you get a score. So you

[10:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=656s) the evaluation you get a score. So you

[10:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=656s) the evaluation you get a score. So you get an aggregate score and there

[10:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=658s) get an aggregate score and there

[10:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=658s) get an aggregate score and there aggregate score could reveal a set of

[11:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=660s) aggregate score could reveal a set of

[11:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=660s) aggregate score could reveal a set of failures. Okay failures could be of any

[11:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=661s) failures. Okay failures could be of any

[11:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=661s) failures. Okay failures could be of any kind. Failures could be hey the read

[11:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=663s) kind. Failures could be hey the read

[11:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=663s) kind. Failures could be hey the read file tool is not working. Sometimes the

[11:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=665s) file tool is not working. Sometimes the

[11:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=665s) file tool is not working. Sometimes the entrance is so slow that like you know

[11:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=667s) entrance is so slow that like you know

[11:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=667s) entrance is so slow that like you know the agent's working it's doing it's

[11:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=669s) the agent's working it's doing it's

[11:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=669s) the agent's working it's doing it's making progress maybe it's running a

[11:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=670s) making progress maybe it's running a

[11:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=670s) making progress maybe it's running a circle but it's so slow that it

[11:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=672s) circle but it's so slow that it

[11:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=672s) circle but it's so slow that it literally like runs out of time it runs

[11:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=673s) literally like runs out of time it runs

[11:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=673s) literally like runs out of time it runs out of like 45 minutes or where many

[11:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=675s) out of like 45 minutes or where many

[11:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=675s) out of like 45 minutes or where many that were given for the task and just

[11:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=677s) that were given for the task and just

[11:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=677s) that were given for the task and just like stop working. So once you do that

[11:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=679s) like stop working. So once you do that

[11:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=679s) like stop working. So once you do that once you understand hey this is what we

[11:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=681s) once you understand hey this is what we

[11:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=681s) once you understand hey this is what we ran this is what the failures we got we

[11:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=683s) ran this is what the failures we got we

[11:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=683s) ran this is what the failures we got we bucket the failures in different

[11:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=684s) bucket the failures in different

[11:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=684s) bucket the failures in different categories. So like 10 failures of you

[11:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=686s) categories. So like 10 failures of you

[11:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=686s) categories. So like 10 failures of you know this kind of tool called 10 10 20

[11:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=688s) know this kind of tool called 10 10 20

[11:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=688s) know this kind of tool called 10 10 20 failures of like say inference not

[11:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=690s) failures of like say inference not

[11:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=690s) failures of like say inference not working or something else right and then

[11:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=691s) working or something else right and then

[11:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=691s) working or something else right and then we once we bucket the failures into

[11:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=693s) we once we bucket the failures into

[11:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=693s) we once we bucket the failures into different categories where like we try

[11:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=694s) different categories where like we try

[11:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=694s) different categories where like we try to make an improvement to the harness

[11:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=696s) to make an improvement to the harness

[11:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=696s) to make an improvement to the harness because like what if there's like

[11:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=698s) because like what if there's like

[11:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=698s) because like what if there's like something wrong with the harness that we

[11:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=699s) something wrong with the harness that we

[11:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=699s) something wrong with the harness that we could change such that the model score

[11:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=701s) could change such that the model score

[11:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=701s) could change such that the model score would improve right and on and on and on

[11:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=703s) would improve right and on and on and on

[11:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=703s) would improve right and on and on and on and you do this iteratively it takes a

[11:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=704s) and you do this iteratively it takes a

[11:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=704s) and you do this iteratively it takes a while it's just like you know hill

[11:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=706s) while it's just like you know hill

[11:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=706s) while it's just like you know hill climbing is like a very groundworky kind

[11:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=708s) climbing is like a very groundworky kind

[11:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=708s) climbing is like a very groundworky kind of process and you know usually usually

[11:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=711s) of process and you know usually usually

[11:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=711s) of process and you know usually usually we we took we used to take a lot of time

[11:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=713s) we we took we used to take a lot of time

[11:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=713s) we we took we used to take a lot of time trying to do this but it was a great way

[11:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=715s) trying to do this but it was a great way

[11:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=715s) trying to do this but it was a great way for us to like learn how to improve

[11:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=716s) for us to like learn how to improve

[11:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=716s) for us to like learn how to improve models. So [snorts]

[11:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=718s) models. So [snorts]

[11:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=718s) models. So [snorts] one way to hill climb on public eval is

[12:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=721s) one way to hill climb on public eval is

[12:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=721s) one way to hill climb on public eval is that like you know we we could have a

[12:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=724s) that like you know we we could have a

[12:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=724s) that like you know we we could have a thing where you know we started doing

[12:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=726s) thing where you know we started doing

[12:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=726s) thing where you know we started doing this thing where we could just like what

[12:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=729s) this thing where we could just like what

[12:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=729s) this thing where we could just like what if we could have like a longunning agent

[12:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=731s) if we could have like a longunning agent

[12:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=731s) if we could have like a longunning agent that runs for a while and automatically

[12:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=733s) that runs for a while and automatically

[12:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=733s) that runs for a while and automatically improves the model score and that's what

[12:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=734s) improves the model score and that's what

[12:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=734s) improves the model score and that's what we did. So we ran GPD 5.6 soul that led

[12:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=738s) we did. So we ran GPD 5.6 soul that led

[12:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=738s) we did. So we ran GPD 5.6 soul that led the work and that was like the main

[12:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=739s) the work and that was like the main

[12:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=739s) the work and that was like the main agent and then uh Ky K3 was just running

[12:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=742s) agent and then uh Ky K3 was just running

[12:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=742s) agent and then uh Ky K3 was just running eval. So what it was doing is that like

[12:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=744s) eval. So what it was doing is that like

[12:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=744s) eval. So what it was doing is that like GBD 5.6 was like hey run these events

[12:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=747s) GBD 5.6 was like hey run these events

[12:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=747s) GBD 5.6 was like hey run these events get these scores let me look at the

[12:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=749s) get these scores let me look at the

[12:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=749s) get these scores let me look at the failures let me look at where where it

[12:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=751s) failures let me look at where where it

[12:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=751s) failures let me look at where where it went wrong and then I'll make changes to

[12:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=753s) went wrong and then I'll make changes to

[12:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=753s) went wrong and then I'll make changes to client's harness and then I'll run these

[12:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=755s) client's harness and then I'll run these

[12:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=755s) client's harness and then I'll run these eval again and then I'll keep trying. So

[12:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=757s) eval again and then I'll keep trying. So

[12:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=757s) eval again and then I'll keep trying. So again, you can see how like very quickly

[12:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=759s) again, you can see how like very quickly

[12:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=759s) again, you can see how like very quickly this can like go off the rails because

[12:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=761s) this can like go off the rails because

[12:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=761s) this can like go off the rails because this is supposed to be like the running

[12:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=762s) this is supposed to be like the running

[12:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=762s) this is supposed to be like the running and evaluation itself takes like an

[12:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=764s) and evaluation itself takes like an

[12:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=764s) and evaluation itself takes like an hour. Then you have to read the

[12:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=765s) hour. Then you have to read the

[12:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=765s) hour. Then you have to read the feedback. Then you have to like see if

[12:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=767s) feedback. Then you have to like see if

[12:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=767s) feedback. Then you have to like see if you're properly interpreting the

[12:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=768s) you're properly interpreting the

[12:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=768s) you're properly interpreting the feedback to make reasonable changes to

[12:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=770s) feedback to make reasonable changes to

[12:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=770s) feedback to make reasonable changes to the harness that are like not reward

[12:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=772s) the harness that are like not reward

[12:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=772s) the harness that are like not reward hacking that are just like actual

[12:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=773s) hacking that are just like actual

[12:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=773s) hacking that are just like actual genuine improvements and then run the

[12:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=774s) genuine improvements and then run the

[12:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=774s) genuine improvements and then run the eval again. So we ran that for 17 hours

[12:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=777s) eval again. So we ran that for 17 hours

[12:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=777s) eval again. So we ran that for 17 hours and uh when we ran that for 17 hours

[13:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=780s) and uh when we ran that for 17 hours

[13:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=780s) and uh when we ran that for 17 hours there was like a legitimate hell

[13:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=781s) there was like a legitimate hell

[13:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=781s) there was like a legitimate hell climbing and what we found that was very

[13:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=783s) climbing and what we found that was very

[13:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=783s) climbing and what we found that was very interesting. This happened with a very

[13:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=785s) interesting. This happened with a very

[13:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=785s) interesting. This happened with a very detailed prompt that accomplished this

[13:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=787s) detailed prompt that accomplished this

[13:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=787s) detailed prompt that accomplished this is that like we had baseline score of

[13:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=790s) is that like we had baseline score of

[13:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=790s) is that like we had baseline score of 27% and then you know we we found issues

[13:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=794s) 27% and then you know we we found issues

[13:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=794s) 27% and then you know we we found issues like loop detection. Loop detection is

[13:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=796s) like loop detection. Loop detection is

[13:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=796s) like loop detection. Loop detection is when a harness just runs in circles

[13:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=797s) when a harness just runs in circles

[13:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=797s) when a harness just runs in circles around a problem right uh we found like

[13:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=800s) around a problem right uh we found like

[13:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=800s) around a problem right uh we found like background process grinding where

[13:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=801s) background process grinding where

[13:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=801s) background process grinding where background processes would run too long.

[13:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=803s) background processes would run too long.

[13:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=803s) background processes would run too long. Uh we found some file index issues and

[13:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=805s) Uh we found some file index issues and

[13:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=805s) Uh we found some file index issues and other things as well. I mean when I say

[13:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=807s) other things as well. I mean when I say

[13:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=807s) other things as well. I mean when I say we what I actually meant is GP 5.6 solve

[13:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=810s) we what I actually meant is GP 5.6 solve

[13:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=810s) we what I actually meant is GP 5.6 solve right? So that found those issues that

[13:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=812s) right? So that found those issues that

[13:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=812s) right? So that found those issues that found and bucketed those issues. It's

[13:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=813s) found and bucketed those issues. It's

[13:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=814s) found and bucketed those issues. It's like hey how about I make those changes

[13:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=816s) like hey how about I make those changes

[13:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=816s) like hey how about I make those changes in the harness and I run the eval again

[13:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=818s) in the harness and I run the eval again

[13:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=818s) in the harness and I run the eval again and it iteratively built the hill

[13:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=819s) and it iteratively built the hill

[13:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=819s) and it iteratively built the hill climbing process completely autonomously

[13:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=821s) climbing process completely autonomously

[13:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=821s) climbing process completely autonomously on its own without my interference and

[13:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=823s) on its own without my interference and

[13:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=823s) on its own without my interference and 13 hours later when I come back to it it

[13:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=825s) 13 hours later when I come back to it it

[13:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=825s) 13 hours later when I come back to it it like it has improved the harness's

[13:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=827s) like it has improved the harness's

[13:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=827s) like it has improved the harness's performance in so many ways. So see what

[13:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=830s) performance in so many ways. So see what

[13:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=830s) performance in so many ways. So see what does that mean? What does that mean? It

[13:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=831s) does that mean? What does that mean? It

[13:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=831s) does that mean? What does that mean? It just it just basically means that like

[13:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=833s) just it just basically means that like

[13:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=834s) just it just basically means that like we're at a point where the experience

[13:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=836s) we're at a point where the experience

[13:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=836s) we're at a point where the experience could make the harness better. That like

[13:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=838s) could make the harness better. That like

[13:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=838s) could make the harness better. That like there's like a two-level learning loop

[14:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=840s) there's like a two-level learning loop

[14:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=840s) there's like a two-level learning loop where the harness does some action,

[14:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=842s) where the harness does some action,

[14:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=842s) where the harness does some action, interacts with the environment,

[14:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=844s) interacts with the environment,

[14:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=844s) interacts with the environment, understands whether or not it did the

[14:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=845s) understands whether or not it did the

[14:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=845s) understands whether or not it did the right thing or the wrong thing and gets

[14:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=847s) right thing or the wrong thing and gets

[14:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=847s) right thing or the wrong thing and gets some reward and and like almost and then

[14:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=850s) some reward and and like almost and then

[14:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=850s) some reward and and like almost and then has there's like a meta agent that's

[14:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=852s) has there's like a meta agent that's

[14:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=852s) has there's like a meta agent that's observing the main harness and finding

[14:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=854s) observing the main harness and finding

[14:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=854s) observing the main harness and finding the changes to it and just like

[14:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=856s) the changes to it and just like

[14:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=856s) the changes to it and just like incorporating those changes and

[14:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=857s) incorporating those changes and

[14:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=857s) incorporating those changes and improving. Now it's I think in my

[14:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=859s) improving. Now it's I think in my

[14:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=859s) improving. Now it's I think in my personal opinion it's much harder to do

[14:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=861s) personal opinion it's much harder to do

[14:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=861s) personal opinion it's much harder to do in real time because if you do it in

[14:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=863s) in real time because if you do it in

[14:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=863s) in real time because if you do it in real time you overindex on solving

[14:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=865s) real time you overindex on solving

[14:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=865s) real time you overindex on solving certain problems or you're just like too

[14:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=868s) certain problems or you're just like too

[14:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=868s) certain problems or you're just like too cartel to a specific thing. But I do

[14:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=870s) cartel to a specific thing. But I do

[14:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=870s) cartel to a specific thing. But I do think that this is something that works

[14:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=871s) think that this is something that works

[14:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=872s) think that this is something that works like on a broader scale of like working

[14:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=873s) like on a broader scale of like working

[14:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=873s) like on a broader scale of like working on a wide range of problems

[14:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=875s) on a wide range of problems

[14:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=876s) on a wide range of problems and you want to be able to like you want

[14:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=878s) and you want to be able to like you want

[14:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=878s) and you want to be able to like you want you want to be able to justify the

[14:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=879s) you want to be able to justify the

[14:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=879s) you want to be able to justify the changes you're making on a broader act

[14:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=882s) changes you're making on a broader act

[14:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=882s) changes you're making on a broader act applicability so that the changes you

[14:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=884s) applicability so that the changes you

[14:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=884s) applicability so that the changes you make here will apply to a broader range

[14:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=886s) make here will apply to a broader range

[14:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=886s) make here will apply to a broader range of problems as well. So what does that

[14:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=889s) of problems as well. So what does that

[14:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=889s) of problems as well. So what does that mean? Like what am I trying to get at

[14:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=892s) mean? Like what am I trying to get at

[14:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=892s) mean? Like what am I trying to get at here? The point I'm trying to get at is

[14:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=894s) here? The point I'm trying to get at is

[14:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=894s) here? The point I'm trying to get at is that the models are so good right now

[14:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=895s) that the models are so good right now

[14:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=895s) that the models are so good right now that they can if they can improve the

[14:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=897s) that they can if they can improve the

[14:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=897s) that they can if they can improve the harness, they can improve a lot of other

[14:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=898s) harness, they can improve a lot of other

[14:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=898s) harness, they can improve a lot of other things. So then what else can they

[14:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=899s) things. So then what else can they

[15:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=900s) things. So then what else can they improve? So this is this is a this is a

[15:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=902s) improve? So this is this is a this is a

[15:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=902s) improve? So this is this is a this is a post uh primarily from OpenI where GP2

[15:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=905s) post uh primarily from OpenI where GP2

[15:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=905s) post uh primarily from OpenI where GP2 5.6 autonomously rewrote and optimized

[15:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=908s) 5.6 autonomously rewrote and optimized

[15:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=908s) 5.6 autonomously rewrote and optimized its own production kernels and those

[15:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=910s) its own production kernels and those

[15:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=910s) its own production kernels and those production kernel improvements remove

[15:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=913s) production kernel improvements remove

[15:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=913s) production kernel improvements remove the end to-end serving costs by 20%. you

[15:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=916s) the end to-end serving costs by 20%. you

[15:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=916s) the end to-end serving costs by 20%. you know it also had 15% higher to token

[15:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=919s) know it also had 15% higher to token

[15:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=919s) know it also had 15% higher to token generation efficiency and this was like

[15:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=920s) generation efficiency and this was like

[15:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=920s) generation efficiency and this was like completely done by 5.6 soul on itself,

[15:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=923s) completely done by 5.6 soul on itself,

[15:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=924s) completely done by 5.6 soul on itself, right? So what that means is that like

[15:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=926s) right? So what that means is that like

[15:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=926s) right? So what that means is that like inference is becoming much cheaper and

[15:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=928s) inference is becoming much cheaper and

[15:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=928s) inference is becoming much cheaper and that means that like if on if if for a

[15:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=930s) that means that like if on if if for a

[15:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=930s) that means that like if on if if for a company like OpenAI if you're having 20%

[15:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=933s) company like OpenAI if you're having 20%

[15:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=933s) company like OpenAI if you're having 20% low lower serving cost for a frontier

[15:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=935s) low lower serving cost for a frontier

[15:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=935s) low lower serving cost for a frontier model that could be a billions of

[15:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=937s) model that could be a billions of

[15:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=937s) model that could be a billions of dollars of improvement in price and you

[15:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=939s) dollars of improvement in price and you

[15:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=939s) dollars of improvement in price and you could you could do that as well for your

[15:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=940s) could you could do that as well for your

[15:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=940s) could you could do that as well for your own models. Um another interesting thing

[15:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=944s) own models. Um another interesting thing

[15:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=944s) own models. Um another interesting thing is that like when people have kernel

[15:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=945s) is that like when people have kernel

[15:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=945s) is that like when people have kernel writing uh competitions so core auto is

[15:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=947s) writing uh competitions so core auto is

[15:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=947s) writing uh competitions so core auto is a company which has a kernel writing

[15:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=949s) a company which has a kernel writing

[15:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=949s) a company which has a kernel writing competitions and you'll notice that like

[15:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=952s) competitions and you'll notice that like

[15:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=952s) competitions and you'll notice that like they they mentioned that like the person

[15:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=954s) they they mentioned that like the person

[15:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=954s) they they mentioned that like the person who won the kernel writing competition

[15:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=956s) who won the kernel writing competition

[15:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=956s) who won the kernel writing competition the explic said that I have never

[15:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=958s) the explic said that I have never

[15:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=958s) the explic said that I have never handwritten a GPU operator it's all ALM

[16:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=961s) handwritten a GPU operator it's all ALM

[16:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=961s) handwritten a GPU operator it's all ALM and agent driven so notice that like how

[16:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=963s) and agent driven so notice that like how

[16:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=963s) and agent driven so notice that like how like the best uh inference improvements

[16:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=966s) like the best uh inference improvements

[16:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=966s) like the best uh inference improvements kernel improvements are at this point

[16:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=968s) kernel improvements are at this point

[16:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=968s) kernel improvements are at this point completely exclusively done by a swarm

[16:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=970s) completely exclusively done by a swarm

[16:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=970s) completely exclusively done by a swarm of agents or generally like agents doing

[16:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=972s) of agents or generally like agents doing

[16:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=972s) of agents or generally like agents doing some magic to you know magically improve

[16:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=975s) some magic to you know magically improve

[16:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=975s) some magic to you know magically improve like the kernel kernel code and have

[16:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=977s) like the kernel kernel code and have

[16:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=977s) like the kernel kernel code and have faster inference cheaper inference. Um

[16:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=980s) faster inference cheaper inference. Um

[16:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=980s) faster inference cheaper inference. Um yeah so basically like the improvement

[16:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=983s) yeah so basically like the improvement

[16:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=983s) yeah so basically like the improvement surface that you think of is not just

[16:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=985s) surface that you think of is not just

[16:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=985s) surface that you think of is not just like the harness improvement it's

[16:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=987s) like the harness improvement it's

[16:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=987s) like the harness improvement it's literally the whole surface stack right

[16:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=988s) literally the whole surface stack right

[16:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=988s) literally the whole surface stack right so the whole stack from like evaluation

[16:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=991s) so the whole stack from like evaluation

[16:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=991s) so the whole stack from like evaluation to harness improvements to training data

[16:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=993s) to harness improvements to training data

[16:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=993s) to harness improvements to training data to post training to world interaction to

[16:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=995s) to post training to world interaction to

[16:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=995s) to post training to world interaction to inference like literally the entire pip

[16:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=998s) inference like literally the entire pip

[16:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=998s) inference like literally the entire pip part of the pipeline like all the way

[16:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=999s) part of the pipeline like all the way

[16:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1000s) part of the pipeline like all the way from like electricity going through your

[16:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1003s) from like electricity going through your

[16:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1003s) from like electricity going through your machine to like making a network request

[16:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1005s) machine to like making a network request

[16:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1005s) machine to like making a network request going through the model models doing

[16:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1007s) going through the model models doing

[16:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1007s) going through the model models doing inference um making a response and

[16:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1009s) inference um making a response and

[16:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1009s) inference um making a response and getting back to you like every single

[16:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1011s) getting back to you like every single

[16:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1011s) getting back to you like every single part of the pipeline we can optimize it.

[16:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1013s) part of the pipeline we can optimize it.

[16:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1013s) part of the pipeline we can optimize it. So the whole RSI loop has like um

[16:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1015s) So the whole RSI loop has like um

[16:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1015s) So the whole RSI loop has like um there's a couple different parts so it's

[16:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1016s) there's a couple different parts so it's

[16:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1016s) there's a couple different parts so it's like pre-training um uh neatronic

[16:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1019s) like pre-training um uh neatronic

[16:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1019s) like pre-training um uh neatronic examples of like you know AI curated uh

[17:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1021s) examples of like you know AI curated uh

[17:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1021s) examples of like you know AI curated uh pre-training data that was good um

[17:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1023s) pre-training data that was good um

[17:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1023s) pre-training data that was good um there's like post- training examples

[17:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1024s) there's like post- training examples

[17:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1024s) there's like post- training examples which are you know where there's like

[17:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1027s) which are you know where there's like

[17:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1027s) which are you know where there's like constitutional AI examples of like post-

[17:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1029s) constitutional AI examples of like post-

[17:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1030s) constitutional AI examples of like post- training uh improvements then inference

[17:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1032s) training uh improvements then inference

[17:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1032s) training uh improvements then inference I gave you the example of GP 5.6 soul

[17:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1034s) I gave you the example of GP 5.6 soul

[17:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1034s) I gave you the example of GP 5.6 soul for eval and eval generation. I give you

[17:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1036s) for eval and eval generation. I give you

[17:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1036s) for eval and eval generation. I give you our own example of how we improved

[17:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1038s) our own example of how we improved

[17:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1038s) our own example of how we improved ourselves on our own eval. Um in general

[17:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1041s) ourselves on our own eval. Um in general

[17:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1041s) ourselves on our own eval. Um in general what you want to think of is that like

[17:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1043s) what you want to think of is that like

[17:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1043s) what you want to think of is that like anytime a model is trying to do

[17:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1044s) anytime a model is trying to do

[17:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1044s) anytime a model is trying to do something and it's like not able to do

[17:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1046s) something and it's like not able to do

[17:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1046s) something and it's like not able to do it or not able to accomplish it. So for

[17:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1048s) it or not able to accomplish it. So for

[17:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1048s) it or not able to accomplish it. So for example like imagine you're having a

[17:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1050s) example like imagine you're having a

[17:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1050s) example like imagine you're having a long conversation where you ask the

[17:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1051s) long conversation where you ask the

[17:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1051s) long conversation where you ask the model something obvious. So like it's

[17:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1053s) model something obvious. So like it's

[17:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1053s) model something obvious. So like it's like or something that you think is not

[17:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1056s) like or something that you think is not

[17:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1056s) like or something that you think is not necessarily obvious but it's like it's

[17:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1058s) necessarily obvious but it's like it's

[17:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1058s) necessarily obvious but it's like it's hard but you would assume that the model

[17:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1060s) hard but you would assume that the model

[17:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1060s) hard but you would assume that the model could potentially do it. So to give an

[17:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1062s) could potentially do it. So to give an

[17:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1062s) could potentially do it. So to give an example, let's say you want to make uh

[17:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1064s) example, let's say you want to make uh

[17:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1064s) example, let's say you want to make uh add a new column to a database and then

[17:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1066s) add a new column to a database and then

[17:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1066s) add a new column to a database and then fill that column, right? And then put a

[17:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1069s) fill that column, right? And then put a

[17:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1069s) fill that column, right? And then put a cache in front of it. So what happens is

[17:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1071s) cache in front of it. So what happens is

[17:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1071s) cache in front of it. So what happens is it does the whole thing, but for some

[17:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1073s) it does the whole thing, but for some

[17:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1073s) it does the whole thing, but for some reason the way it pushes the cache in

[17:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1074s) reason the way it pushes the cache in

[17:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1074s) reason the way it pushes the cache in front of the whole uh database like it

[17:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1076s) front of the whole uh database like it

[17:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1076s) front of the whole uh database like it kind of doesn't do it right. So at that

[17:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1077s) kind of doesn't do it right. So at that

[17:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1078s) kind of doesn't do it right. So at that point you're like, "Oh no, set this

[17:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1079s) point you're like, "Oh no, set this

[18:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1080s) point you're like, "Oh no, set this cache differently. Set this as the TTL."

[18:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1081s) cache differently. Set this as the TTL."

[18:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1082s) cache differently. Set this as the TTL." And then you give like a correction

[18:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1083s) And then you give like a correction

[18:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1083s) And then you give like a correction signal and then that kind of like fixes

[18:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1085s) signal and then that kind of like fixes

[18:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1085s) signal and then that kind of like fixes your problem and you kind of go about

[18:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1087s) your problem and you kind of go about

[18:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1087s) your problem and you kind of go about your day.

[18:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1088s) your day.

[18:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1088s) your day. The way I want to think of it is that I

[18:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1089s) The way I want to think of it is that I

[18:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1089s) The way I want to think of it is that I want to think of like every single time

[18:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1091s) want to think of like every single time

[18:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1091s) want to think of like every single time like a human correction to the digital

[18:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1094s) like a human correction to the digital

[18:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1094s) like a human correction to the digital landscape is a part of a delta that

[18:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1098s) landscape is a part of a delta that

[18:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1098s) landscape is a part of a delta that needs to be covered. And what that means

[18:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1101s) needs to be covered. And what that means

[18:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1101s) needs to be covered. And what that means is that like every single time you're

[18:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1102s) is that like every single time you're

[18:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1102s) is that like every single time you're finding like the edge of model

[18:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1104s) finding like the edge of model

[18:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1104s) finding like the edge of model capabilities of like how far it can push

[18:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1106s) capabilities of like how far it can push

[18:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1106s) capabilities of like how far it can push for infants, how much it can push for

[18:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1107s) for infants, how much it can push for

[18:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1107s) for infants, how much it can push for this, how much it can push for, you

[18:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1109s) this, how much it can push for, you

[18:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1109s) this, how much it can push for, you know, solving your problems. You want to

[18:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1112s) know, solving your problems. You want to

[18:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1112s) know, solving your problems. You want to find that delta and we want to like push

[18:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1114s) find that delta and we want to like push

[18:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1114s) find that delta and we want to like push that frontier a little bit further. a

[18:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1116s) that frontier a little bit further. a

[18:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1116s) that frontier a little bit further. a little bit further and that's what post

[18:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1118s) little bit further and that's what post

[18:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1118s) little bit further and that's what post training does, right? So post training

[18:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1119s) training does, right? So post training

[18:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1119s) training does, right? So post training is is this is in that post training we

[18:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1122s) is is this is in that post training we

[18:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1122s) is is this is in that post training we find these like really interesting and

[18:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1125s) find these like really interesting and

[18:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1125s) find these like really interesting and really hard long horizon problems for uh

[18:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1127s) really hard long horizon problems for uh

[18:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1127s) really hard long horizon problems for uh for coding agents or for other kinds of

[18:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1128s) for coding agents or for other kinds of

[18:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1128s) for coding agents or for other kinds of problems and then just like slowly hill

[18:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1130s) problems and then just like slowly hill

[18:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1130s) problems and then just like slowly hill climb their models on on those kind of

[18:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1132s) climb their models on on those kind of

[18:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1132s) climb their models on on those kind of problems. So anything that has a

[18:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1134s) problems. So anything that has a

[18:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1134s) problems. So anything that has a verifiable loop can be optimized but is

[18:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1138s) verifiable loop can be optimized but is

[18:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1138s) verifiable loop can be optimized but is this is not just limited to software

[19:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1140s) this is not just limited to software

[19:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1140s) this is not just limited to software like literally like the physical world

[19:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1142s) like literally like the physical world

[19:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1142s) like literally like the physical world itself could have a verifiable loop. So

[19:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1144s) itself could have a verifiable loop. So

[19:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1144s) itself could have a verifiable loop. So to give an example like let me let me

[19:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1146s) to give an example like let me let me

[19:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1146s) to give an example like let me let me give you a very funny kind of bizarre

[19:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1148s) give you a very funny kind of bizarre

[19:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1148s) give you a very funny kind of bizarre example where like it's like you you

[19:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1151s) example where like it's like you you

[19:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1151s) example where like it's like you you look at this and u you um

[19:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1155s) look at this and u you um

[19:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1155s) look at this and u you um um look at this uh like robot and it's

[19:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1158s) um look at this uh like robot and it's

[19:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1158s) um look at this uh like robot and it's it's typing something and it's like

[19:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1160s) it's typing something and it's like

[19:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1160s) it's typing something and it's like really really like interesting what the

[19:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1162s) really really like interesting what the

[19:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1162s) really really like interesting what the robot is typing.

[19:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1172s) So notice how like notice how the robot

[19:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1174s) So notice how like notice how the robot

[19:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1174s) So notice how like notice how the robot is itself typing like how it wants to

[19:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1177s) is itself typing like how it wants to

[19:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1177s) is itself typing like how it wants to like solve robotics and imagine if the

[19:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1179s) like solve robotics and imagine if the

[19:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1179s) like solve robotics and imagine if the robot itself could like interpret the

[19:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1181s) robot itself could like interpret the

[19:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1181s) robot itself could like interpret the results of of like the answers and just

[19:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1184s) results of of like the answers and just

[19:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1184s) results of of like the answers and just like keep improving then that means that

[19:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1186s) like keep improving then that means that

[19:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1186s) like keep improving then that means that like this is you know basically like an

[19:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1188s) like this is you know basically like an

[19:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1188s) like this is you know basically like an ASI robot. Another example that I could

[19:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1190s) ASI robot. Another example that I could

[19:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1190s) ASI robot. Another example that I could give you from my uh personal life is

[19:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1193s) give you from my uh personal life is

[19:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1193s) give you from my uh personal life is that like um I have this um chip that

[19:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1196s) that like um I have this um chip that

[19:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1196s) that like um I have this um chip that has the ability to see and has a speaker

[19:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1199s) has the ability to see and has a speaker

[19:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1199s) has the ability to see and has a speaker and has a microphone inside of it. So I

[20:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1200s) and has a microphone inside of it. So I

[20:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1200s) and has a microphone inside of it. So I was trying to set up this uh with my

[20:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1203s) was trying to set up this uh with my

[20:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1203s) was trying to set up this uh with my machine and it was um a gift that I had

[20:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1207s) machine and it was um a gift that I had

[20:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1207s) machine and it was um a gift that I had for my girlfriend. And uh what I found

[20:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1209s) for my girlfriend. And uh what I found

[20:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1209s) for my girlfriend. And uh what I found very interesting about this is as I was

[20:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1211s) very interesting about this is as I was

[20:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1211s) very interesting about this is as I was trying to set it up, it was very slow.

[20:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1213s) trying to set it up, it was very slow.

[20:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1213s) trying to set it up, it was very slow. It was slow to respond. It had higher

[20:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1214s) It was slow to respond. It had higher

[20:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1214s) It was slow to respond. It had higher latency. It had an agent in and of

[20:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1216s) latency. It had an agent in and of

[20:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1216s) latency. It had an agent in and of itself. Right? So what I did was I

[20:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1218s) itself. Right? So what I did was I

[20:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1218s) itself. Right? So what I did was I connected the step to my um computer and

[20:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1222s) connected the step to my um computer and

[20:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1222s) connected the step to my um computer and I let um I let um coding agents run on

[20:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1227s) I let um I let um coding agents run on

[20:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1227s) I let um I let um coding agents run on this hardware and basically run on this

[20:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1229s) this hardware and basically run on this

[20:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1229s) this hardware and basically run on this hardware automatically make tests to it.

[20:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1232s) hardware automatically make tests to it.

[20:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1232s) hardware automatically make tests to it. So like my computer would talk to my

[20:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1235s) So like my computer would talk to my

[20:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1235s) So like my computer would talk to my hardware give it some tests like hey do

[20:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1238s) hardware give it some tests like hey do

[20:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1238s) hardware give it some tests like hey do this do that and just like what do you

[20:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1239s) this do that and just like what do you

[20:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1239s) this do that and just like what do you see right now? So then it would notice

[20:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1242s) see right now? So then it would notice

[20:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1242s) see right now? So then it would notice the end to end latency of like the

[20:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1244s) the end to end latency of like the

[20:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1244s) the end to end latency of like the inference response of like how fast uh

[20:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1246s) inference response of like how fast uh

[20:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1246s) inference response of like how fast uh my hardware is responding and then um

[20:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1249s) my hardware is responding and then um

[20:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1249s) my hardware is responding and then um what I found is that like it was able to

[20:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1251s) what I found is that like it was able to

[20:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1251s) what I found is that like it was able to do the breakdown of like every single

[20:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1253s) do the breakdown of like every single

[20:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1253s) do the breakdown of like every single case and scenario where there was like

[20:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1255s) case and scenario where there was like

[20:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1255s) case and scenario where there was like high latency um slower response or or

[20:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1259s) high latency um slower response or or

[20:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1259s) high latency um slower response or or just the algorithm wasn't right and I

[21:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1261s) just the algorithm wasn't right and I

[21:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1261s) just the algorithm wasn't right and I could then set up a recursive loop where

[21:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1265s) could then set up a recursive loop where

[21:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1265s) could then set up a recursive loop where both the firmware could be updated the

[21:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1267s) both the firmware could be updated the

[21:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1267s) both the firmware could be updated the um the tool calls the inference model

[21:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1270s) um the tool calls the inference model

[21:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1270s) um the tool calls the inference model used um the speech to text model used

[21:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1272s) used um the speech to text model used

[21:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1272s) used um the speech to text model used all of those could be improved

[21:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1273s) all of those could be improved

[21:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1273s) all of those could be improved automatically with like an OpenRouter

[21:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1275s) automatically with like an OpenRouter

[21:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1275s) automatically with like an OpenRouter key which I'd given to a coding agent

[21:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1277s) key which I'd given to a coding agent

[21:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1277s) key which I'd given to a coding agent and then overnight with self-running

[21:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1279s) and then overnight with self-running

[21:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1279s) and then overnight with self-running experiments with the hardware uh my um

[21:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1282s) experiments with the hardware uh my um

[21:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1282s) experiments with the hardware uh my um this tiny little piece of beautiful

[21:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1284s) this tiny little piece of beautiful

[21:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1284s) this tiny little piece of beautiful hardware that I have was like deeply

[21:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1286s) hardware that I have was like deeply

[21:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1286s) hardware that I have was like deeply optimized so was this also like even

[21:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1288s) optimized so was this also like even

[21:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1288s) optimized so was this also like even hardware could be a part of the RSI loop

[21:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1290s) hardware could be a part of the RSI loop

[21:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1290s) hardware could be a part of the RSI loop so what happens when you connect these

[21:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1292s) so what happens when you connect these

[21:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1292s) so what happens when you connect these loops right that's the next question

[21:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1293s) loops right that's the next question

[21:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1293s) loops right that's the next question like you could have cheaper inference

[21:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1295s) like you could have cheaper inference

[21:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1295s) like you could have cheaper inference you could have like more experiments

[21:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1296s) you could have like more experiments

[21:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1296s) you could have like more experiments budget you have like you know better

[21:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1298s) budget you have like you know better

[21:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1298s) budget you have like you know better data and better models. So, is it

[21:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1300s) data and better models. So, is it

[21:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1300s) data and better models. So, is it singularity? It's a very question to ask

[21:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1303s) singularity? It's a very question to ask

[21:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1303s) singularity? It's a very question to ask and I think that the part of the parts

[21:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1304s) and I think that the part of the parts

[21:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1304s) and I think that the part of the parts of the loop are already there, right?

[21:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1306s) of the loop are already there, right?

[21:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1306s) of the loop are already there, right? Like we're we we are really we really

[21:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1308s) Like we're we we are really we really

[21:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1308s) Like we're we we are really we really are very close to parts of the RSI loop,

[21:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1311s) are very close to parts of the RSI loop,

[21:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1311s) are very close to parts of the RSI loop, right? And what do we do like what do we

[21:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1313s) right? And what do we do like what do we

[21:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1314s) right? And what do we do like what do we do about this and it's like how do we

[21:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1315s) do about this and it's like how do we

[21:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1315s) do about this and it's like how do we think of that in terms of like our

[21:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1317s) think of that in terms of like our

[21:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1317s) think of that in terms of like our career and how do you think of that in

[21:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1319s) career and how do you think of that in

[21:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1319s) career and how do you think of that in terms of you know our lives and what do

[22:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1320s) terms of you know our lives and what do

[22:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1320s) terms of you know our lives and what do we want to do next? So here's my advice

[22:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1322s) we want to do next? So here's my advice

[22:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1322s) we want to do next? So here's my advice to everyone who is watching this like

[22:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1324s) to everyone who is watching this like

[22:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1324s) to everyone who is watching this like please please please please please move

[22:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1326s) please please please please please move

[22:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1326s) please please please please please move closer to the RS right so it's like

[22:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1329s) closer to the RS right so it's like

[22:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1329s) closer to the RS right so it's like right now like people are working on all

[22:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1331s) right now like people are working on all

[22:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1331s) right now like people are working on all these like different kinds of software

[22:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1332s) these like different kinds of software

[22:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1332s) these like different kinds of software which is like you know a agents for you

[22:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1334s) which is like you know a agents for you

[22:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1334s) which is like you know a agents for you know construction agents for this agents

[22:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1335s) know construction agents for this agents

[22:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1336s) know construction agents for this agents for that and it's like I think that

[22:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1337s) for that and it's like I think that

[22:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1337s) for that and it's like I think that you're you know what the way I see those

[22:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1341s) you're you know what the way I see those

[22:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1341s) you're you know what the way I see those things is that like I think of those as

[22:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1342s) things is that like I think of those as

[22:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1342s) things is that like I think of those as like you're missing the forest for the

[22:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1344s) like you're missing the forest for the

[22:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1344s) like you're missing the forest for the for the trees you really are and it's

[22:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1345s) for the trees you really are and it's

[22:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1345s) for the trees you really are and it's like you have this like very interesting

[22:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1349s) like you have this like very interesting

[22:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1349s) like you have this like very interesting godlike technology that can change parts

[22:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1351s) godlike technology that can change parts

[22:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1351s) godlike technology that can change parts of humanity. And then we're not

[22:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1353s) of humanity. And then we're not

[22:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1354s) of humanity. And then we're not ambitious enough to pursue the best

[22:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1355s) ambitious enough to pursue the best

[22:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1355s) ambitious enough to pursue the best problems. We really are not. And we're

[22:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1357s) problems. We really are not. And we're

[22:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1357s) problems. We really are not. And we're just like I think that it's not just

[22:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1359s) just like I think that it's not just

[22:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1359s) just like I think that it's not just like the problem isn't just that we're

[22:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1361s) like the problem isn't just that we're

[22:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1361s) like the problem isn't just that we're not ambitious and we're also like what

[22:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1363s) not ambitious and we're also like what

[22:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1363s) not ambitious and we're also like what we're pursuing. A lot of those things

[22:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1366s) we're pursuing. A lot of those things

[22:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1366s) we're pursuing. A lot of those things are just like one model release away

[22:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1367s) are just like one model release away

[22:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1367s) are just like one model release away from like being completely like wiped

[22:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1369s) from like being completely like wiped

[22:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1369s) from like being completely like wiped out, right? So like it's also like it's

[22:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1371s) out, right? So like it's also like it's

[22:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1371s) out, right? So like it's also like it's also like it may not be that

[22:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1373s) also like it may not be that

[22:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1373s) also like it may not be that intellectually simulating because like

[22:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1375s) intellectually simulating because like

[22:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1375s) intellectually simulating because like all you're essentially all we're

[22:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1376s) all you're essentially all we're

[22:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1376s) all you're essentially all we're essentially doing is like sitting all

[22:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1377s) essentially doing is like sitting all

[22:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1377s) essentially doing is like sitting all day. It's like hey cloud do this for me.

[22:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1378s) day. It's like hey cloud do this for me.

[22:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1378s) day. It's like hey cloud do this for me. Hey cloud do this for me. And I don't

[23:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1380s) Hey cloud do this for me. And I don't

[23:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1380s) Hey cloud do this for me. And I don't know if that's like I don't know if

[23:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1381s) know if that's like I don't know if

[23:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1381s) know if that's like I don't know if that's like the most intellectual

[23:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1382s) that's like the most intellectual

[23:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1382s) that's like the most intellectual stimulating or like or like um

[23:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1385s) stimulating or like or like um

[23:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1385s) stimulating or like or like um thoughtful use of a person's time like

[23:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1387s) thoughtful use of a person's time like

[23:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1387s) thoughtful use of a person's time like maybe maybe you know we're supposed to

[23:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1388s) maybe maybe you know we're supposed to

[23:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1388s) maybe maybe you know we're supposed to be solving harder and more uh more

[23:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1390s) be solving harder and more uh more

[23:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1390s) be solving harder and more uh more frontier problems and I think that like

[23:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1392s) frontier problems and I think that like

[23:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1392s) frontier problems and I think that like it's like the ceiling is like our

[23:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1393s) it's like the ceiling is like our

[23:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1393s) it's like the ceiling is like our ambition. So instead of like working on

[23:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1395s) ambition. So instead of like working on

[23:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1395s) ambition. So instead of like working on these kind of things my suggestion to

[23:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1397s) these kind of things my suggestion to

[23:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1397s) these kind of things my suggestion to people is to like you know do things

[23:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1400s) people is to like you know do things

[23:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1400s) people is to like you know do things that of course like goes without saying

[23:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1402s) that of course like goes without saying

[23:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1402s) that of course like goes without saying but you should say things you believe in

[23:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1403s) but you should say things you believe in

[23:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1403s) but you should say things you believe in and you know I'm just a guy on the

[23:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1404s) and you know I'm just a guy on the

[23:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1404s) and you know I'm just a guy on the internet but do things that help you

[23:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1407s) internet but do things that help you

[23:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1407s) internet but do things that help you move closer to the RSI loop right? So

[23:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1409s) move closer to the RSI loop right? So

[23:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1409s) move closer to the RSI loop right? So like anything that just like the end

[23:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1411s) like anything that just like the end

[23:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1411s) like anything that just like the end to-end pipeline of like hey this is

[23:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1414s) to-end pipeline of like hey this is

[23:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1414s) to-end pipeline of like hey this is where like the models are lacking like

[23:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1416s) where like the models are lacking like

[23:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1416s) where like the models are lacking like it's like whether that's inference

[23:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1417s) it's like whether that's inference

[23:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1417s) it's like whether that's inference whether that's post training whether

[23:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1419s) whether that's post training whether

[23:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1419s) whether that's post training whether that's literally like physical

[23:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1420s) that's literally like physical

[23:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1420s) that's literally like physical interaction whether that could be you

[23:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1422s) interaction whether that could be you

[23:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1422s) interaction whether that could be you know send sending end to-end pipelines

[23:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1424s) know send sending end to-end pipelines

[23:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1424s) know send sending end to-end pipelines of truly any kind you know whether

[23:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1426s) of truly any kind you know whether

[23:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1426s) of truly any kind you know whether that's hardware problems whether that's

[23:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1428s) that's hardware problems whether that's

[23:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1428s) that's hardware problems whether that's um networking problems whether that's um

[23:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1430s) um networking problems whether that's um

[23:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1430s) um networking problems whether that's um building a sense of community things

[23:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1431s) building a sense of community things

[23:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1431s) building a sense of community things that just you know um things that just

[23:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1434s) that just you know um things that just

[23:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1434s) that just you know um things that just like help um you as an engineer get

[23:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1437s) like help um you as an engineer get

[23:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1437s) like help um you as an engineer get closer to the benefit benefits of the

[23:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1439s) closer to the benefit benefits of the

[23:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1439s) closer to the benefit benefits of the improvements um in uh in u um in in the

[24:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1444s) improvements um in uh in u um in in the

[24:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1444s) improvements um in uh in u um in in the model. So this is a very interesting

[24:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1445s) model. So this is a very interesting

[24:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1445s) model. So this is a very interesting slide from Sasha topic. So um um they

[24:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1450s) slide from Sasha topic. So um um they

[24:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1450s) slide from Sasha topic. So um um they mentioned that like you want to have

[24:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1453s) mentioned that like you want to have

[24:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1453s) mentioned that like you want to have like this you're in this space where

[24:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1455s) like this you're in this space where

[24:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1455s) like this you're in this space where it's like you know you only you only

[24:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1457s) it's like you know you only you only

[24:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1457s) it's like you know you only you only have like 1 to five years left to

[24:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1458s) have like 1 to five years left to

[24:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1458s) have like 1 to five years left to contribute to luxury to the world what

[24:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1459s) contribute to luxury to the world what

[24:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1459s) contribute to luxury to the world what do you do right so it's like um they

[24:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1461s) do you do right so it's like um they

[24:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1461s) do you do right so it's like um they they give great suggestions where like

[24:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1463s) they give great suggestions where like

[24:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1463s) they give great suggestions where like prefer ambitious targeted and past

[24:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1465s) prefer ambitious targeted and past

[24:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1465s) prefer ambitious targeted and past collaborations right and you know in

[24:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1468s) collaborations right and you know in

[24:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1468s) collaborations right and you know in general like the time scale like all the

[24:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1470s) general like the time scale like all the

[24:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1470s) general like the time scale like all the timelines that you give to yourself like

[24:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1471s) timelines that you give to yourself like

[24:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1472s) timelines that you give to yourself like they should be just like completely

[24:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1473s) they should be just like completely

[24:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1473s) they should be just like completely squeezed out should be completely

[24:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1474s) squeezed out should be completely

[24:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1474s) squeezed out should be completely squeezed out because it's like a lot of

[24:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1475s) squeezed out because it's like a lot of

[24:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1475s) squeezed out because it's like a lot of times and you also have to think through

[24:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1477s) times and you also have to think through

[24:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1477s) times and you also have to think through that like if if the model if the problem

[24:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1479s) that like if if the model if the problem

[24:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1479s) that like if if the model if the problem that you're working on is too ambitious

[24:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1480s) that you're working on is too ambitious

[24:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1480s) that you're working on is too ambitious like it's like is it probably is it

[24:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1483s) like it's like is it probably is it

[24:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1483s) like it's like is it probably is it probably better to wait for just like

[24:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1484s) probably better to wait for just like

[24:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1484s) probably better to wait for just like the next next frontier model to come out

[24:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1486s) the next next frontier model to come out

[24:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1486s) the next next frontier model to come out or not like that's that's something you

[24:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1488s) or not like that's that's something you

[24:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1488s) or not like that's that's something you also have to weigh through right and uh

[24:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1491s) also have to weigh through right and uh

[24:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1491s) also have to weigh through right and uh you you ideally want to work on problems

[24:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1493s) you you ideally want to work on problems

[24:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1493s) you you ideally want to work on problems that benefit from scale and you know you

[24:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1496s) that benefit from scale and you know you

[24:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1496s) that benefit from scale and you know you also at the same time want to avoid

[24:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1497s) also at the same time want to avoid

[24:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1497s) also at the same time want to avoid projects that can be solved by sale

[24:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1498s) projects that can be solved by sale

[24:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1498s) projects that can be solved by sale alone because then you know if even if

[25:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1500s) alone because then you know if even if

[25:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1500s) alone because then you know if even if you like it's like if you're working on

[25:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1501s) you like it's like if you're working on

[25:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1501s) you like it's like if you're working on a problem and then the next model

[25:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1503s) a problem and then the next model

[25:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1503s) a problem and then the next model release comes out it solves your problem

[25:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1504s) release comes out it solves your problem

[25:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1504s) release comes out it solves your problem so that's not that's probably not a good

[25:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1505s) so that's not that's probably not a good

[25:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1505s) so that's not that's probably not a good thing to do so it's very like you have

[25:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1507s) thing to do so it's very like you have

[25:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1507s) thing to do so it's very like you have to like think philosophically of like

[25:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1509s) to like think philosophically of like

[25:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1509s) to like think philosophically of like what is the endeavor that benefits uh

[25:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1512s) what is the endeavor that benefits uh

[25:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1512s) what is the endeavor that benefits uh greatly from auto improvements but like

[25:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1513s) greatly from auto improvements but like

[25:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1514s) greatly from auto improvements but like isn't also completely wiped out from it

[25:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1515s) isn't also completely wiped out from it

[25:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1516s) isn't also completely wiped out from it and then ideally like it's like if the

[25:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1518s) and then ideally like it's like if the

[25:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1518s) and then ideally like it's like if the if the endeavor could also uh be a part

[25:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1520s) if the endeavor could also uh be a part

[25:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1520s) if the endeavor could also uh be a part of the the like the improvement pipeline

[25:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1524s) of the the like the improvement pipeline

[25:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1524s) of the the like the improvement pipeline uh that could be amazing. So you see a

[25:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1526s) uh that could be amazing. So you see a

[25:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1526s) uh that could be amazing. So you see a lot of companies raising on say um

[25:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1529s) lot of companies raising on say um

[25:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1529s) lot of companies raising on say um making data pipelines and there's a lot

[25:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1531s) making data pipelines and there's a lot

[25:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1531s) making data pipelines and there's a lot of money there. There's a a lot of

[25:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1533s) of money there. There's a a lot of

[25:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1534s) of money there. There's a a lot of companies working on physical hardware

[25:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1535s) companies working on physical hardware

[25:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1535s) companies working on physical hardware and stuff. So I think if you're a

[25:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1536s) and stuff. So I think if you're a

[25:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1536s) and stuff. So I think if you're a software engineer watching these, it's

[25:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1537s) software engineer watching these, it's

[25:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1538s) software engineer watching these, it's like I think that if you're if you feel

[25:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1540s) like I think that if you're if you feel

[25:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1540s) like I think that if you're if you feel like you're like here the delta of your

[25:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1542s) like you're like here the delta of your

[25:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1542s) like you're like here the delta of your intellect and ability in your

[25:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1544s) intellect and ability in your

[25:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1544s) intellect and ability in your day-to-day, it's like not contributing

[25:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1546s) day-to-day, it's like not contributing

[25:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1546s) day-to-day, it's like not contributing as in like you have the intellect, you

[25:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1549s) as in like you have the intellect, you

[25:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1549s) as in like you have the intellect, you can make a difference. But like if cloud

[25:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1550s) can make a difference. But like if cloud

[25:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1550s) can make a difference. But like if cloud is just like oneshotting all those

[25:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1552s) is just like oneshotting all those

[25:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1552s) is just like oneshotting all those problems like in your day-to-day and

[25:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1554s) problems like in your day-to-day and

[25:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1554s) problems like in your day-to-day and you're just like kind of like becoming a

[25:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1555s) you're just like kind of like becoming a

[25:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1555s) you're just like kind of like becoming a me proxy, you should work to think

[25:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1556s) me proxy, you should work to think

[25:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1556s) me proxy, you should work to think through. Take a step back and think

[25:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1557s) through. Take a step back and think

[25:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1557s) through. Take a step back and think through. like okay where are the

[25:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1559s) through. like okay where are the

[25:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1559s) through. like okay where are the problems where where you're actually

[26:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1561s) problems where where you're actually

[26:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1561s) problems where where you're actually truly uh contributing and that they will

[26:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1563s) truly uh contributing and that they will

[26:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1563s) truly uh contributing and that they will stand the test of time and then that

[26:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1565s) stand the test of time and then that

[26:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1565s) stand the test of time and then that kind of reflection and realignment right

[26:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1567s) kind of reflection and realignment right

[26:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1567s) kind of reflection and realignment right now would I think like have like very

[26:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1569s) now would I think like have like very

[26:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1569s) now would I think like have like very interesting consequences for you um in

[26:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1572s) interesting consequences for you um in

[26:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1572s) interesting consequences for you um in the long term um so yeah so thank you so

[26:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1575s) the long term um so yeah so thank you so

[26:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1575s) the long term um so yeah so thank you so much you know thank you so much for your

[26:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1577s) much you know thank you so much for your

[26:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1577s) much you know thank you so much for your time like I think that uh I think that

[26:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1579s) time like I think that uh I think that

[26:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1579s) time like I think that uh I think that if you're watching this and if you've

[26:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1580s) if you're watching this and if you've

[26:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1580s) if you're watching this and if you've gone this far I think that you're

[26:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1582s) gone this far I think that you're

[26:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1582s) gone this far I think that you're definitely contemplating like one of the

[26:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1584s) definitely contemplating like one of the

[26:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1584s) definitely contemplating like one of the most interesting improvements in the

[26:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1586s) most interesting improvements in the

[26:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1586s) most interesting improvements in the course of humanity And uh you know

[26:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1588s) course of humanity And uh you know

[26:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1588s) course of humanity And uh you know there's many philosophical arguments

[26:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1590s) there's many philosophical arguments

[26:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1590s) there's many philosophical arguments here. Some people might say hey you know

[26:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1592s) here. Some people might say hey you know

[26:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1592s) here. Some people might say hey you know I I think I'm maybe I'm off on the

[26:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1595s) I I think I'm maybe I'm off on the

[26:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1595s) I I think I'm maybe I'm off on the timelines or maybe maybe I'm I'm just

[26:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1597s) timelines or maybe maybe I'm I'm just

[26:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1597s) timelines or maybe maybe I'm I'm just being too pessimistic or maybe I'm being

[26:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1599s) being too pessimistic or maybe I'm being

[26:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1599s) being too pessimistic or maybe I'm being too optimistic. But I would love to hear

[26:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1601s) too optimistic. But I would love to hear

[26:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1601s) too optimistic. But I would love to hear all those thoughts. So this is you know

[26:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1602s) all those thoughts. So this is you know

[26:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1602s) all those thoughts. So this is you know my QR code. Please feel free to scan it

[26:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1604s) my QR code. Please feel free to scan it

[26:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1604s) my QR code. Please feel free to scan it and let me know all your thoughts. Uh

[26:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1606s) and let me know all your thoughts. Uh

[26:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1606s) and let me know all your thoughts. Uh thank you so much. Very kind of you to

[26:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1607s) thank you so much. Very kind of you to

[26:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1607s) thank you so much. Very kind of you to give me your time.