# Devin 2.0 and the Future of SWE

Scott Wu

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/MI83buT_23o-devin-2-0-future-swe) · [Video](https://www.youtube.com/watch?v=MI83buT_23o)

## Devin 2.0 and the Future of SWE

As coding agents take on longer tasks, the hard problem shifts from following instructions to understanding repositories, clarifying intent, and testing their own work.

Familiarity with repositories, pull requests, linting, and continuous integration will help you follow the engineering examples.

### How much work can an agent do before you intervene?

[0:43](https://www.youtube.com/watch?v=MI83buT_23o&t=43s)

Ask **GPT-3** to do something, and it might produce only a few acceptable words before you need to redirect it. **GPT-3.5** and **GPT-4** extend that useful stretch. Scott Wu opens with this practical measure of capability: **how much work can an agent complete before a human must intervene or steer it?** For software, the progression runs from individual lines and functions toward tasks representing hours of human work.

Wu cites roughly seven-month doubling for agent task length generally and seventy-day doubling for coding tasks. These are trend estimates, not a universal coding law or a Devin-specific performance measurement. The distinction between task length and runtime matters: related [METR research](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) measures the human duration of tasks an agent can complete at a specified success probability, primarily 50%, rather than elapsed agent runtime before intervention. Its broader estimate already covers software and reasoning tasks; the faster SWE-bench Verified trend depends on human-time estimates that assume repository familiarity.

![Slide showing a SWE-bench Verified task time-horizon chart with model markers, error bars, and an upward trend line.](https://ai.engineer/api/talk-essay-media/68431712e4ac9ab1f3306a91544ea246b053e3d447e1f9f68a5110a49cb5636b)

Coding task complexity solvable by AI doubles every 70 days.

Wu translates four to six annual doublings into a claimed 16–64× increase in coding work capacity per year over the preceding couple of years. The product change is easier to recognize than the multiplier: in his account, eighteen months earlier, tab completion was the only coding experience with product-market fit. The useful interaction was to supply the current file and predict the next line. Delegating an engineering task asks for something much larger.

**Longer tasks change both the engineering bottleneck and the interface.** A tool that predicts a line needs different support from one that investigates a bug, and an agent that can investigate a bug needs different support from one that takes responsibility for a backlog. Wu describes the period from Cognition’s late-2023 beginnings through June 2025 as a sequence of these capability tiers, each arriving on roughly the timescale of a few months.

### Repetitive migrations: follow the procedure, retain the corrections

[3:31](https://www.youtube.com/watch?v=MI83buT_23o&t=211s)

The first broadly adopted use case Wu describes is repetitive migration: JavaScript to TypeScript, an Angular upgrade, or a Java version upgrade. A large repository needs the same procedure applied file by file. Upgrade documentation may spell out the steps, but applying them still requires judgment about the code in front of you. The task is explicit without necessarily being reducible to one deterministic transformation.

![Repetitive migrations slide describing single-file edits, limited context, static rules, and programmatic verification.](https://ai.engineer/api/talk-essay-media/a261460ff5e2ad65f2aa2754a31d318780ecd60710644555a9d9f42c906d0c4b)

Repetitive migrations apply the same rules across thousands of files.

That combination made migrations a useful reliability target. An agent might occasionally solve something harder, but occasional success is not enough when a procedure must work repeatedly across, in Wu’s illustrative example, ten thousand files. Repetition also makes the work tedious for humans. Delegating it frees attention for more creative engineering while giving the agent a bounded procedure to execute.

The immediate capability requirement was **instruction following**. Cognition built [Playbooks](https://docs.devin.ai/product-guides/creating-playbooks) so a user could spell out a sequence of steps and have Devin follow it. That fits migration work particularly well; much of software engineering cannot be specified as a fixed sequence with no decisions left to make.

Repeated execution then exposed a second requirement: **persistent feedback**. If a human corrects the same omission on one migration, that correction should inform the next one. Wu describes Knowledge or memory as retaining instructions such as remembering a particular action whenever a certain situation appears. Playbooks supply the procedure; memory carries forward what people learn while applying it. He notes that Cognition subsequently rebuilt this memory system multiple times.

### Small fixes still need a working repository

[6:22](https://www.youtube.com/watch?v=MI83buT_23o&t=382s)

By late summer and early fall, the useful task expanded from repeated procedures to contained bugs and features. Wu’s example is a repository selector: put the currently selected repositories at the top of the dropdown instead of leaving checked items scattered throughout it. The desired behavior is clear, but the agent must find and change the relevant implementation.

For a TypeScript implementation of that behavior, a stable partition keeps the existing order within each group and leaves the repository records unchanged:

```typescript
type Repository = {
  id: string;
  selected: boolean;
};

function selectedFirst(repositories: readonly Repository[]): Repository[] {
  return [
    ...repositories.filter(repo => repo.selected),
    ...repositories.filter(repo => !repo.selected),
  ];
}

const repositories: Repository[] = [
  { id: "api", selected: false },
  { id: "web", selected: true },
  { id: "docs", selected: false },
];

const ordered = selectedFirst(repositories);
// ordered IDs: web, api, docs
// repositories remains in its original order.
```

This makes the small change concrete: ordering changes, while identity and selection state are preserved.

Wu compares this scope to work you might assign an intern, typically involving one or two files. Yet a small diff still needs a functioning development environment. Devin must be able to set up the repository, run lint, and run CI checks to obtain basic evidence that its change works.

The infrastructure response was a prepared repository environment with **snapshots** that could be loaded again or rolled back. A clean remote VM supplied the place to execute the linter and CI checks. This broadened Devin’s value beyond a specialized migration tool: it could become a junior collaborator for ordinary, contained engineering work.

### Broader bugs require relationships, not just source text

[8:14](https://www.youtube.com/watch?v=MI83buT_23o&t=494s)

Later in the fall, broader bugs and requests introduced a different problem. A report might identify a symptom without identifying its cause. The agent must investigate, determine which files participate in the behavior, and coordinate a change that may span hundreds of lines. Reading the file containing the symptom is no longer enough.

Wu points to several ways to treat a repository as a connected system:

- **Call hierarchies and language servers** expose relationships between definitions and uses.
- **Git commit history** helps explain how files relate through earlier changes.
- **Lint and cross-file references** provide additional context for checking and coordinating edits.

The goal is consistency across the affected files. An individually plausible edit can still be wrong if the corresponding changes elsewhere are missing.

Once broader issues became practical to delegate, Slack became a natural interface: it was already where teams discussed bugs and requested work. A person could tag Devin in the discussion and ask it to investigate or build something. Wu connects this capability tier with [Devin’s general availability](https://cognition.com/blog/devin-generally-available), when a newly configured agent could take on a wider range of tasks.

The underlying requirement resembles onboarding a human engineer. Someone on their first day may understand the language but lack the repository-specific knowledge needed to choose a safe change. Devin likewise needed to build a representation of the codebase, use it to plan the task, identify all affected files, and then produce the diff. **Delegation became useful as codebase understanding improved.**

### Explore together before handing off the task

[10:41](https://www.youtube.com/watch?v=MI83buT_23o&t=641s)

By spring, more difficult tasks exposed an ambiguity on the human side: the person assigning the work might not yet know exactly what should change. Improving architecture requires understanding its constraints. Speeding up a slow function requires profiling. Better error handling requires enumerating possible failures and deciding what each should do. A short prompt cannot fully specify decisions that have not been made.

[DeepWiki](https://cognition.com/blog/deepwiki) made Devin’s codebase understanding useful to the human as well. Wu describes it as exposing a representation Devin already needed internally, so people could understand the repository and ask questions about it. Search supported the closely related activity of investigating a particular part of the codebase before committing to a change.

The workflow became iterative:

1. Ask questions about the relevant code and behavior.
2. Explore the repository with the agent until the intended change is clear.
3. Define the task, then send the agent off to implement it.

The Search slide shows that handoff explicitly: a codebase answer sits beside source code, followed by a prompt requesting implementation and unit tests. Exploration supplies the context for execution.

![Slide showing a codebase search answer beside source code, with a red-outlined prompt requesting implementation and unit tests.](https://ai.engineer/api/talk-essay-media/d3e8ec155f733524a94981da712f2be81cd4350388690625d9341989fcf0a1b6)

Devin Search enables handoff to the agent with the right context.

This alternation also shaped the [Devin 2.0](https://cognition.com/blog/devin-2) IDE experience. The human could work closely with Devin at the points requiring judgment, then let it continue independently. Wu illustrates the workflow with 10–20% of a task closely monitored and the remaining 80–90% performed autonomously; these are example proportions, not measured supervision requirements.

### A backlog adds routing and escalation decisions

[12:51](https://www.youtube.com/watch?v=MI83buT_23o&t=771s)

Wu’s June tier is handing over many backlog tasks at once. This combines the earlier capabilities rather than replacing them. Linear and Jira provide the tickets, but integrating with a tracker is only the entry point: the agent still has to determine what each ticket means and whether the task is sufficiently specified. June marks the capability tier in this retrospective, not the first release of every supporting integration.

In an organization with multiple repositories, even choosing where to work becomes part of the assignment. Devin must identify the right repository, locate the relevant subsystem, and coordinate edits across files. Before doing that autonomously, it also needs to distinguish two states: enough confidence in the task to proceed, or enough uncertainty to ask a human for clarification or approval. **Knowing when to stop and ask is part of useful autonomy.**

### Complete pull requests need an iterative testing loop

[14:04](https://www.youtube.com/watch?v=MI83buT_23o&t=844s)

Asynchronous work raises the standard for what the agent should deliver. If it is returning an entire pull request, particularly for a larger task, it needs to test its own changes rather than hand the human an unchecked diff. The Ticket to PR slide makes this concrete with a checklist labeled “Local testing complete” above an application screenshot.

![Ticket to PR slide with a testing checklist labeled Local testing complete above an application screenshot.](https://ai.engineer/api/talk-essay-media/3521bfcb3ad81db4073409d91e4a473eb8ce69c4e277da5d5a1475beeefac5bb)

Testing reduces the overhead of parallel tasks for asynchronous agents.

The agent needs an iterative loop: run the code in its local execution environment, examine the result, and use that feedback to continue the work. But an environment capable of running commands solves only part of the problem. The agent must also know **what to test and what to look for**. Choosing the relevant behavior and interpreting the result require substantial task and codebase context. Testing becomes a reasoning problem as well as an execution capability.

### The next doubling asks a different engineering question

[14:34](https://www.youtube.com/watch?v=MI83buT_23o&t=874s)

The next question is whether an agent can take on an entire project rather than an individual task. Wu presents that as the next frontier, not an established capability demonstrated here. From a distance, every doubling may look like another increment on the same curve. In practice, each increase introduces different work.

Single-line completion could be treated as text prediction over the current file. Longer engineering work requires participation in the human workflow through Linear, Slack, and Jira; accepting feedback and steering; and helping people plan before implementation begins. It also requires execution tooling for autonomous tests, longer-term decisions, debugging generated outputs, and shell commands that gather useful feedback. These are distinct systems that must work together for a longer assignment to succeed.

Wu closes by forecasting another 16–64× increase in coding-agent work capacity over the following twelve months. That is his forecast, not a verified outcome. The engineering challenge behind it is concrete: each larger unit of delegated work requires new ways to establish intent, obtain feedback, and decide what to do next.

## Resources

- [METR's task-completion time horizon research](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) — Explains human-duration task horizons, success thresholds, exponential trends, and limits of extrapolation.
- [Measuring AI Ability to Complete Long Software Tasks](https://papers.neurips.cc/paper_files/paper/2025/file/85069585133c4c168c865e65d72e9775-Paper-Conference.pdf) — The NeurIPS 2025 paper details the evaluation methodology and the approximately 70-day SWE-bench Verified trend.
- [Devin's general availability announcement](https://cognition.com/blog/devin-generally-available) — The December 2024 release describes Slack delegation, first-draft pull requests, and practical task-scoping guidance.
- [Devin 2.0 launch](https://cognition.com/blog/devin-2) — Introduces interactive planning, parallel cloud IDE sessions, Devin Search, and repository wikis.
- [DeepWiki: AI docs for any repo](https://cognition.com/blog/deepwiki) — Explains the public repository documentation and codebase question-answering service built from Devin Wiki and Search.
- [Creating Devin Playbooks](https://docs.devin.ai/product-guides/creating-playbooks) — Current guidance for reusable task instructions, procedures, success criteria, and iterative improvement.
- [Devin's 2024 release history](https://docs.devin.ai/release-notes/2024) — Dated records of Playbooks, Knowledge, snapshots, machine access, and Slack workflow changes.
- [Devin's 2025 release history](https://docs.devin.ai/release-notes/2025) — Tracks interactive planning, DeepWiki, Linear and Jira integration, and confidence-based approval behavior.

## Slides

### Building Devin: 18 Months of Lessons Learned

Scott Wu, CEO of Cognition AI, introduces lessons from 18 months building Devin.

[0:23](https://www.youtube.com/watch?v=MI83buT_23o&t=23s) [Title slide naming Scott Wu, CEO of Cognition AI.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc00fc1d134dcbc79f2ca2ea6f29c9155755871025b762600676e8cc6a68937b.webp)

Building Devin: 18 Months of Lessons Learned

Scott Wu
CEO, Cognition AI

Interface text:
AIE
AI Engineer World’s Fair
daily
Microsoft
aws
smol ai

### Moore’s Law for AI agents

METR plots task length at a 50% success rate against model release date, showing a roughly seven-month doubling time. Scott Wu frames agent capacity as how much work it can do before intervention.

[0:49](https://www.youtube.com/watch?v=MI83buT_23o&t=49s) [A METR chart shows AI task length at a 50% success rate increasing with model release date, with an estimated seven-month doubling time.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b820f5a6949b4385cb714c11ddad9897942a24508ddfa20a64d8286154aceb6.webp)

AGENTS
Moore’s Law for AI agents

The length of tasks AIs can do is doubling every 7 months
Task length (at 50% success rate)
METR

Y-axis ticks: 1 sec, 4 sec, 15 sec, 1 min, 4 min, 15 min, 1 hr, 4 hrs
Timeline: 2020, 2022, 2024, 2026
X-axis: Model release date

Task examples: Answer question; Count words in passage; Find fact on web; Train classifier; Optimize code for custom chip
Model labels: GPT 2, GPT 3, GPT 3.5, GPT 4, GPT 4o, o1, Sonnet 3.7

CC-BY
metr.org
Source: Measuring AI Ability to Complete Long Tasks

Interface text: AIE; AI Engineer World's Fair; daily; Microsoft; aws; smol ai

### Coding task complexity doubles every 70 days

Using data from January 1, 2023 onward, the fitted SWE-Bench Verified task horizon doubles every 70 days: newer models reach a 50% success rate on tasks that take humans longer.

[1:27](https://www.youtube.com/watch?v=MI83buT_23o&t=87s) [A SWE-Bench Verified chart plots model task time horizons against release date and reports a 70-day doubling time.](https://assets.ai.engineer/sites/aiecode2025/sha256/83f27b6143f678d1d4b18e250e84b54cad12a175954316d9764213207b0fb3ea.webp)

AGENTS
Coding task complexity solvable by AI doubles every 70 days

Time Horizon for SWE-Bench Verified Tasks

Legend:
Claude 3.5 Sonnet (New)
Claude 3.5 Sonnet (Old)
Claude 3 Opus
o1
GPT-4o
GPT-4 1106

Y-axis: Task time (for humans) that model completes with 50% success rate
Y-axis ticks: 4 sec, 8 sec, 15 sec, 30 sec, 1 min, 2 min, 4 min, 8 min, 15 min, 30 min, 1 hr, 2 hrs, 4 hrs
X-axis: Model release date
Timeline: 2023, 2024, 2025

Doubling time: 70 days
2023-01-01+ data
R²: 0.87

Source: Measuring AI Ability to Complete Long Tasks

Interface text: AIE; AI Engineer World's Fair; Microsoft; aws; smol ai

### From text completion to AI engineers

The speaker describes a shift from tab completion to AI engineers that implement tasks, illustrated by a collage of Devin pull-request notifications.

[2:30](https://www.youtube.com/watch?v=MI83buT_23o&t=150s) [A slide contrasts an earlier code-completion example with a collage of AI engineering work notifications.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c3e08b90572530932d89a8f3a35daaabfb37afdb14e60923b2e9c7d426c265a.webp)

Cognition

18 Months Ago
Text completion

per_token_kls = (teacher_lo[illegible]
masked_kls = per_token_kls
if avg_over_sequence:

Today
AI Engineers

Interface text:
Devin (Staging) APP 6:59 PM
PR PR #6708: Refactor markdown conversion to [illegible]
formatting in usacognition/devin-webapp wa[illegible]
@Dev Chheda
View Devin Session

Devin (Staging) APP 7:21 PM
PR PR #6857: Fix curs[illegible]
merged by andrewgcodes (requested [illegible]
View D[illegible]

Devin (Staging) APP
PR PR #6851: Ad[illegible]
merged by dev-cogni
View Devin Session

Devin (Staging) APP 10:30 PM
PR PR #4500: Implement round-robin scheduling between organi[illegible]
queue in usacognition/devin was merged by walnutwaldo (requested [illegible]
View Devin Session

Devin (Staging) APP
PR PR #188: Update [illegible]
walnutwaldo (requested [illegible]
View Devin Session

Devin (Staging) APP 3:00 AM
PR PR #6506: Fix arrow[illegible]
usacognition/devin-webapp [illegible]
View Devin Session

Devin (Staging) APP 2:17 PM
PR PR #6635: Add atta[illegible]
webapp was merged by jac[illegible]
View Devin Session

Devin (Staging)
PR PR #65[illegible]
merged by Joseph-Gross (requested by @Joseph Gros[illegible]
View Devin Session

AIE
AI Engineer World's Fair
Graphite
Microsoft
aws
smol ai

### Evolution of coding-agent tasks

Cognition’s task tiers progress from repetitive migrations through isolated fixes, broad requests, and underspecified tasks to ticket-to-PR work across the 2024–2025 timeline.

[3:11](https://www.youtube.com/watch?v=MI83buT_23o&t=191s) [Timeline from repetitive migrations through isolated fixes, broader requests, underspecified tasks, and ticket-to-PR work.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c7d38f29eaaac4234f997c85ad7b2632a183f106732ec1f601ec2c2a8b9394c.webp)

Cognition

Timeline: 2024, 2025

Repetitive migrations
Isolated fixes
Broad bugs & requests
Underspecified tasks
Ticket to PR

Interface text:
AIE
AI Engineer
World’s Fair
Graphit[illegible]
Microsoft
aws
smol ai

### Repetitive migrations

Repetitive migrations apply the same rules file by file, with limited context and programmatic checks. Examples include JavaScript-to-TypeScript migrations and version upgrades.

[3:30](https://www.youtube.com/watch?v=MI83buT_23o&t=210s) [Slide describes migrating files using repeated rules, limited context, and programmatic verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c82493070611f90cbf35ee8e4c9739332dcfc5afbca979245544d52b36b9a6d.webp)

May ’24
01 Repetitive migrations
Cognition

Migrate thousands of files one at a time by repeatedly applying the same rules

Requires editing a single file and viewing only a handful of files for context

Rules are static across task instances

There is a way to programmatically verify the work (lint, dummy data pipeline, ...)

Interface text:
AIE
AI Engineer World’s Fair
vellum
Microsoft
aws
smol ai

### Repeatable tasks and playbooks

Playbooks give Devin a clear sequence of steps for repetitive migrations. The example lists prerequisites and setup steps for moving data from a relational database to MongoDB.

[5:12](https://www.youtube.com/watch?v=MI83buT_23o&t=312s) [Slide pairs a statement about step-by-step instructions with a migration playbook example.](https://assets.ai.engineer/sites/aiecode2025/sha256/88c3daf7825b3a4738ecd917672af0db2c293530c7721cba66a1053cd14d7ea7.webp)

May ’24
01 Repetitive migrations
Cognition

Execute repeatable tasks reliably with step-by-step instructions

Migrating Data from Relational Database to MongoDB Using Mongo Syphon
Playbook: Migrating Data from Relational Database to MongoDB Using Mongo Syphon

Overview
This playbook provides detailed steps to migrate data from a running relational db instance inside a Docker container to MongoDB Cloud using Mongo Syphon.

What's Needed from the User
• Relational db connection details (host, port, username, password, database name) in case of existing database credentials within the system as docker container or cloud service
• MongoDB connection details (username, password, database name) in case of existing database credentials within the system as docker container or cloud service

Procedure
1. Set Up Mongo Syphon
  • Download Mongo Syphon:
    • Visit the Mongo Syphon GitHub repository and download the latest release.
    • Extract the downloaded package to a desired location on the machine.

Interface text:
AIE
AI Engineer World's Fair
vellum
Microsoft
aws
smol ai

### Isolated fixes

Devin took on isolated, intern-level bugs and features through Slack, such as moving selected repositories to the top of a dropdown.

[6:26](https://www.youtube.com/watch?v=MI83buT_23o&t=386s) [Slide describes small bug fixes and features, with an example request in a messaging interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce34c757dc0eb3b76df5867259c39b640c85f8c8b2db5b38f51a262736761950.webp)

August ’24
02 Isolated fixes
Cognition

Fix bugs and implement features that can be described in a few sentences

Triggered from Slack to delegate incoming requests and bug reports
Work on the same repos every time
Intern-level complexity tasks

Interface text:
Bryce Hunt  6:59 PM
@Devin (Staging) in the devin search repos select dropdown please list the currently selected ones at the top
9 replies  Last reply 5 days ago

AIE
AI Engineer World’s Fair
vellum
Microsoft
aws
smol ai

### Prepare repositories for agents

Set up the repository ahead of time so Devin can run checks such as lint and CI. The speaker also describes snapshots that can be reloaded or rolled back.

[7:26](https://www.youtube.com/watch?v=MI83buT_23o&t=446s) [Slide recommends setting up a repository in advance and shows a setup interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/40fcc4202e1132099586a336977231c1a8a18f5f23bc1956ce0acdb140c027c8.webp)

August ’24
02 Isolated fixes
Cognition

Set up repository ahead of time to minimize friction for the agent

Interface text:
Set up austrian-code-wizard/c3po
Devin’s Machine
Devin’s Browser
Video Guide
Documentation
1. Git Pull
2. Configure Secrets (Optional)
3. Install Dependencies
4. Maintain Dependencies
How should we keep dependencies up to date?
These commands will be run after git pull during session startup in case new dependencies were added.
pip install -r requirements.txt
Add
Skip
Verify
Docs
5. Setup Lint
6. Setup Tests (Optional)
7. Setup Local App (Optional)
8. Additional Notes
Finish
Finish Setup Later
Devin IDE
Import local VS Code settings
Go To File
Quick Agent (Beta)
PROBLEMS
OUTPUT
DEBUG CONSOLE
TERMINAL
[illegible]

AIE
AI Engineer World’s Fair
daily
Microsoft
aws
smol ai

### Instant startup and snapshots

Devin’s repository setup can be saved as a snapshot, then reloaded or rolled back. The remote VM can run checks such as CI and lint.

[7:41](https://www.youtube.com/watch?v=MI83buT_23o&t=461s) [Slide describing instant startup and snapshots alongside a repository workspace screenshot.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ab465419478efc6e8126fde31ca1d30ca9f8ec24d8d322c9545c7976bf2ad3b.webp)

August ’24
02 Isolated fixes

Build instant startup + snapshots for a smooth remote experience

Interface text:
Settings > Devin’s Machine
Devin’s Machine
Manage the machine and repositories on Devin’s Workspace. Devin uses a new copy of the machine for each session. Learn m[illegible]

Machine Contents
Find a repository...
+ Repository
c3po · Python
c3po-website · HTML
opencircles · TypeScript
python-docker · Python
spring2024-assignment5-alignment · Python

Add a New Repository to Devin’s Workspace
Find a repository...
austrian-code-wizard/WebShop
[NeurIPS 2022] WebShop: Towards Scalable Real-World Web Interaction...
Python
Last updated 1 year ago

austrian-code-wizard/polynomial
Python
Last updated 1 year ago

austrian-code-wizard/miniwob-plusplus
MiniWoB++: a web interaction benchmark for reinforcement learning
HTML
Last updated 2 years ago

austrian-code-wizard/dreamgrader-alpha-centauri
Giving Feedback on Interactive Student Programs with Meta-Exploration...
Python
Last updated 2 years ago

austrian-code-wizard/alpha-centauri
Last updated 2 years ago

austrian-code-wizard/exo-hunt
Jupyter Notebook
Last updated 3 years ago

austrian-code-wizard/bux-bear-frontend
JavaScript
Last updated 3 years ago

austrian-code-wizard/bux-bear-backend
TypeScript
Last updated 3 years ago

AIE
AI Engineer World’s Fair
daily
Microsoft
aws
smol ai

### Broad bugs and requests

Broader fixes required tracing control and data flow across files, often changing hundreds of lines. The example shows a merged PR for round-robin scheduling in a repository indexing queue.

[8:12](https://www.youtube.com/watch?v=MI83buT_23o&t=492s) [Slide describing refactors and fixes that require understanding relationships across a codebase.](https://assets.ai.engineer/sites/aiecode2025/sha256/4cda58abfa0b2c8a23c1cfeea62a038833aec73bad942d610267803151b823e4.webp)

November ’24
03 Broad bugs & requests
Cognition

Refactors, feature development, and bug fixes that require understanding complex relationships in the codebase

Requires understanding control and data flow across multiple codebase locations
Implementation involves interdependent changes across files
Hundreds of lines of code changed

Interface text:
Devin (Staging) APP 10:30 PM
🎉 PR PR #4500: Implement round-robin scheduling between organizations in repo indexing job queue in usacognition/devin was merged by walnutwaldo (requested by @Walden) 🚀
View Devin Session

AIE
AI Engineer World’s Fair
daily
Microsoft
aws
smol ai

### Planning across the codebase

Devin plans changes across the codebase before writing a diff. The example maps a logging callback across four training algorithms, then proposes a short test.

[9:38](https://www.youtube.com/watch?v=MI83buT_23o&t=578s) [Slide explaining upfront planning beside an example implementation plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/1869c91aebacffe63a85db13061487ea9fe5ba9886340df6f85bd55442fad0cc.webp)

November ’24
03 Broad bugs & requests
Cognition

Plan upfront to better understand connections between relevant parts of the codebase

Plan Overview:
• Add new TrainingLoggingCallback class to utils.py that provides comprehensive training progress logging, metrics tracking, and algorithm-specific information using HuggingFace's TrainerCallback system
• Integrate the callback across all four training algorithms (DPO, SFT, LCDPO, SFT_weighted) in train.py by updating the callback lists in trainer initialization sections
• Update imports in train.py to include the new TrainingLoggingCallback
• Test by running a short training session to verify logging works correctly across algorithms, then commit to existing branch devin/1748894722-add-training-loop-logging and create PR to main

Confidence: High
I will now proceed with executing the plan.

Interface text:
Show detailed plan
AIE
AI Engineer World’s Fair
Graphi[illegible]
Microsoft
aws
smol ai

### Underspecified tasks

As tasks get harder, users may not know everything they want upfront. Working with Devin helps them specify more detail and choose among multiple correct solutions.

[10:38](https://www.youtube.com/watch?v=MI83buT_23o&t=638s) [Slide describing complex tasks that require human assistance and user preferences.](https://assets.ai.engineer/sites/aiecode2025/sha256/394162ac59b3a22cfdcdc32128b48240b7f4166b6a91b818bce3fb65a10360b5.webp)

February ’25
04 Underspecified tasks
Cognition

Complete more complex tasks with human assistance

Users lack sufficient context to write effective prompts
Multiple correct solutions – requires user preference
Small to medium-sized tickets

Interface text:
AIE
AI Engineer World’s Fair
Graphi[illegible]
Microsoft
aws
smol ai

### DeepWiki and codebase understanding

DeepWiki uses repository metadata to understand codebases. The speaker says it lets people inspect Devin’s internal codebase representation and quickly ask questions about it.

[11:39](https://www.youtube.com/watch?v=MI83buT_23o&t=699s) [DeepWiki repository search above a statement about using repository metadata to understand codebases.](https://assets.ai.engineer/sites/aiecode2025/sha256/765d9a82708433a27918f1da7d6cdfd50048dafd860b9ecf04e521271a94a991.webp)

February ’25
04 Underspecified tasks
Cognition

DeepWiki uses repo metadata to gain a deeper understanding of the codebase

Interface text:
DeepWiki
Try DeepWiki for private repos in Devin
Share
Which repo would you like to understand?
Search for repositories (or paste a link)
Add repo
microsoft / vscode
Visual Studio Code
170.1k
mark3labs / mcp-go
A Go implementation of the Model Context Protocol (MCP), enabling seamless integration between LLM...
3.4k
antiwork / gumroad
5.2k
langchain-ai / local-deep-researcher
Fully local web research and report writing assistant
7.0k
meta-llama / llama-models
Utilities intended for use with Llama models.
6.8k

### Devin Search handoff

A Search answer explains the C3PO loss and points to its code. The draft follow-up asks Devin to implement TP and write unit tests, carrying the investigation into an agent task.

[12:04](https://www.youtube.com/watch?v=MI83buT_23o&t=724s) [A full-screen Search answer and code view illustrate handing context to an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fa095788c1d67bb3fcdcd9b583084862b976035e9597dee7f96131fdea514b7.webp)

February ’25
04 Underspecified tasks

Devin Search enables handoff to the agent with the right context

Interface text:
what is the c3po loss formula?
Fast
Searched across c3po
Go deeper

You're asking about the C3PO loss formula, which refers to the loss function used in the Locally Constrained DPO (LCDPO) algorithm implemented in the C3PO codebase.

C3PO Loss Formula

The C3PO loss formula combines three components in the LocallyConstrainedDPOTrainer.compute_loss method lcdpo.py:152-194:

Combined Loss Formula:
loss = dpo_loss + sigma_soft * kd_soft_loss + sigma_hard * kd_hard_loss

Loss Components
1. DPO Loss: Standard Direct Preference Optimization loss from the base DPOTrainer lcdpo.py:167-168
2. Soft Knowledge Distillation Loss: KL divergence between student and teacher models on soft negative examples lcdpo.py:171-172
3. Hard Knowledge Distillation Loss: KL divergence between student and teacher models on hard negative examples lcdpo.py:174-176

Knowledge Distillation Loss Calculation

Each knowledge distillation component is computed using lcdpo.py:71-121:

distillation_loss = masked_dl_div(student_logits, teacher_logits, attentio[illegible]
final_kd_loss = (1 - kd_lambda) * student_target_loss + kd_lambda * distill[illegible]

The masked_dl_div function computes KL divergence while respecting attention masks lcdpo.py:12-29.

Configurable Parameters

The loss is controlled by several hyperparameters defined in TrainingArguments utils.py:87-93:
- lcdpo_lambda: Controls balance between distillation and [illegible]
- lcdpo_temp: Temperature for knowledge distillation
- lcdpo_sigma_soft: Weight for soft negative examples
- lcdpo_sigma_hard: Weight for hard negative examples

This loss function enables the model to learn from preferences (DPO) while maintaining [illegible]

austrian-code-wizard/c3po  src/lcdpo.py

from transformers import PreTrainedModel

def masked_dl_div(pred_logits: torch.Tensor, teacher_logits: [illegible]
    """Compute the KL divergence between two distributions, [illegible]

    Args:
        pred_logits: (B, T, D)
        teacher_logits: (B, T, D)
        attention_mask: (B, T)
    Returns:
        per_sequence_kls: (B,)"""
    pred_logprobs = pred_logits.log_softmax(-1) # (B, T, D)
    teacher_logprobs = teacher_logits.log_softmax(-1) # (B, T, [illegible]
    per_token_kls = (teacher_logprobs.exp() * (teacher_logpro[illegible]
    masked_kls = per_token_kls * (attention_mask if attention[illegible]
    if avg_over_sequence:
        per_sequence_kls = masked_kls.sum(-1) / (attention_ma[illegible]
    else:
        per_sequence_kls = masked_kls.sum(-1) # (B,)
    return per_sequence_kls.mean(0) # scalar

def masked_lm_loss(

super().__init__(*args, **kwargs)

def compute_knowledge_distillation_loss(
    self,
    model: Union[PreTrainedModel, nn.Module],
    batch: Dict[str, Union[List, torch.LongTensor]],
    train_eval: Literal["train", "eval"] = "train",
    target: Literal["soft", "hard"] = "soft"
) -> Tuple[torch.FloatTensor, Dict[str, float]]:
    """Compute the knowledge distillation loss for a batc[illegible]
    Adapted from https://huggingface.co/docs/transformers[illegible]
    """
    metrics = {}

    # Only keep relevant data
    batch = {
        "input_ids": batch[f"{target}_negative_input_ids"[illegible]

Based on this info, please implement TP for the loss function & write unit tests
Deep
Cmd + Enter to construct a Devin prompt
Enter to ask a follow-up

### Ticket to PR

Devin aims to clear medium-sized tickets in parallel within existing workflows. That requires scoping tasks, finding the right code, and deciding when to ask a human for help.

[12:48](https://www.youtube.com/watch?v=MI83buT_23o&t=768s) [Slide describes clearing medium-sized tickets within engineering workflows with minimal interaction and parallel tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/79b23395df6d2b966eb65aebf3d284ef723ce6997d955072e2418f8788f19990.webp)

June ’25
05 Ticket to PR
Cognition

Autonomously clear the user’s backlog of medium-sized tickets

Slot into existing engineering workflows
Minimal human interaction required
Run many tasks at once to multiply productivity

Interface text:
AIE
AI Engineer World’s Fair
vellum
Microsoft
aws
smol ai

### Seeking human feedback

The speaker says agents should ask for help when they cannot confidently understand a task. The example presents a proposed Jira fix and invites feedback before starting a session to create a PR.

[13:40](https://www.youtube.com/watch?v=MI83buT_23o&t=820s) [Slide pairs a statement about seeking feedback when necessary with a ticket-scoping example.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dbe2626b12918de545fc3dd19571dd8bff8888cc6c86d4ecda893493cf632fa.webp)

June ’25
05 Ticket to PR

Proactively seek human feedback only when necessary

Started scoping Linear ticket. Follow along and follow up in Linear.

Hey, I'm Devin! I'm currently researching your task and I'll be back soon with a proposal. Please check back in 5 minutes.

You can track my progress in Devin Space. Later, you will be able to use this Devin Space to start a Devin session to create a PR, or to perform follow-up analysis.

Note: Ticket scoping is still in beta. External links are not yet supported.

The Task: Fix the Jira integration section to display the connected organization name after "Connected to". Confidence: High

Existing Code: The JiraSetupWidget component attempts to display site name from jiraStatusData?.integration?.config?.site_name but it's not showing up. Confidence: High

Proposed Solution: Verify the API response and fix the frontend rendering. Confidence: Medium

Looks good? Click here to start a Devin session to create a PR
Need to explore further? Open in Devin Spaces for follow-up analysis
Needs work? Add clarifications in the issue text or the comments, and @Devin to get an updated proposal.

Interface text:
bugs-frontend > FRONTEN-36
Accept
Mark as duplicate
Links
Activity
Subscribe
stevenhao created the issue from Slack · 15d ago
Slack thread connected in #ux-audit 15d ago
Devin (Staging) 15d ago via Slack
Leave a reply...
devinstaging 15d ago
AIE
AI Engineer World's Fair
vellum
Microsoft
aws
smol ai

### Testing asynchronous agents

A pull request records local checks for a wiki refresh button. The speaker says agents need an iterative testing loop: run the code locally, know what to test, and inspect the results.

[14:00](https://www.youtube.com/watch?v=MI83buT_23o&t=840s) [Slide explains how testing reduces the overhead of parallel agent tasks, alongside a pull request testing example.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f6e51acbe2b236d055e64f216618fa3968a83245d45ad09bff0d96ab7c0339b.webp)

June ’25
05 Ticket to PR

Testing makes asynchronous agents powerful by reducing the overhead of parallel tasks

Testing
Local testing completed:
- Webapp runs successfully on localhost:3000
- Logged in with Google authentication
- Navigated to wiki page for devin-webapp repository
- Confirmed refresh button appears for wiki indexed on "9 May 2025" (>20 hours old)
- Verified button shows "Refreshing..." state when clicked
- Confirmed API call is made to /ada/index_public_repo endpoint

Interface text:
Cognition
Merged
Add refresh button to devin wiki when >20 hours old #7630
[illegible] merged 4 commits into main from [illegible] · 11 hours ago

Embedded wiki screenshot:
Overview
System Architecture
High-Level Architecture
Monorepo Structure
On this page
[illegible]

AIE
AI Engineer World’s Fair
vellum
Microsoft
aws
smol ai

### Beyond?

The speaker asks how coding agents might move from individual tasks to entire projects, where they would need to incorporate human feedback, test their work and make longer-term decisions.

[14:33](https://www.youtube.com/watch?v=MI83buT_23o&t=873s) [Cognition slide labeled September ’25 and 06 Beyond, with “Beyond?” centered on a dark background.](https://assets.ai.engineer/sites/aiecode2025/sha256/bba52f32058d9e52643edf6a253ae11a5045eae8a9919991930c08962a62c5c3.webp)

September ’25
06 Beyond
Cognition

Beyond?

Interface text:
AIE
AI Engineer World’s Fair
vellum
Microsoft
aws
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=MI83buT_23o&t=0s) [upbeat music] Yeah. Well, thank you guys so much for having me.

[0:16](https://www.youtube.com/watch?v=MI83buT_23o&t=16s) It's exciting to be back. It's, uh, I, I was last here at AI Engineer one year ago. Um, and it's kinda crazy. I, I've always been-- I, I've been telling swyx that we need to have these conferences way more often if it's gonna be about AI software engineering.

[0:28](https://www.youtube.com/watch?v=MI83buT_23o&t=28s) Probably should be, like, every two months or something like that [chuckles] with the pace that everything's done. But, but, but gonna be fun to, to talk a little bit about, um, you know, what we've seen in the space and, and what we've learned over the last twelve or eighteen months, uh, building Devin over this time.

[0:43](https://www.youtube.com/watch?v=MI83buT_23o&t=43s) And I wanna start this off with, um, Moore's law for AI agents. And so you can kinda think of the, the, the capability or the capacity of an AI by how much work it can do uninter-uninterrupted until you have to come in and step in and intervene or steer it or whatever it is, right?

[1:02](https://www.youtube.com/watch?v=MI83buT_23o&t=62s) And, um, you know, in GPT-3, for example, it's if you were to go and ask GPT-3 to do something, you know, it could probably get through a few words or so, and then it'll say something where it's like, "Okay, you know, this is probably not the right thing to say." [chuckles]

[1:14](https://www.youtube.com/watch?v=MI83buT_23o&t=74s) Um, and GPT-3.5 was better, and GPT-4 was better, right? Um, and, and so people talk about these lengths of tasks, and what you see in general is that that doubling time is about every seven months, which already is pretty crazy, actually.

[1:27](https://www.youtube.com/watch?v=MI83buT_23o&t=87s) But in code, it's actually even faster. It's every seventy days, which is two or three months. And so, you know, if you look at various software engineering tasks that start from the simplest single functions or single lines, and you go all the way to, you know, we're doing tasks now that take hours of humans' time, and, and

[1:46](https://www.youtube.com/watch?v=MI83buT_23o&t=106s) an AI agent is able to just do all of that, right? Um, and if you think about doubling every seventy days, I mean, basically, you know, every two to three months means you get four to six doublings every year.

[1:56](https://www.youtube.com/watch?v=MI83buT_23o&t=116s) Um, which means [chuckles] that the amount of work that an AI agent can do in code goes something between sixteen and sixty-four x in a year, every year, at least for the last couple years that we've seen.

[2:08](https://www.youtube.com/watch?v=MI83buT_23o&t=128s) Um, and it's kinda crazy to think about, but, but it-- that sounds about right, actually, for, for what we've seen. You know, eighteen months ago, I would say the only really-- the only product experience that had PMF in code was just tab completion, right?

[2:21](https://www.youtube.com/watch?v=MI83buT_23o&t=141s) It was just, like, here's what I have so far. Predict the next line for me. That was kind of all you really could do, um, in, in a way that really worked.

[2:29](https://www.youtube.com/watch?v=MI83buT_23o&t=149s) And we've gone from that, obviously, to full AI engineer that goes and just do-does, does all these tasks for you, right, and implements a ton of these things. And people ask all the time, what is the, um, you know, what, what, what is the, the future interface or what is the right way to do this or what

[2:46](https://www.youtube.com/watch?v=MI83buT_23o&t=166s) are the most important capabilities to solve for? And I think funnily enough, the answer to all these questions actually is it changes every two or three months. Like, every time you get to the next tier, the, the, the bottleneck that you're running into or the most important capability or the right way you should be interfacing with it,

[3:02](https://www.youtube.com/watch?v=MI83buT_23o&t=182s) like, all these actually change at, at each point. And so I wanted to talk a bit about some of the, the, those tiers for us over the last year or so.

[3:13](https://www.youtube.com/watch?v=MI83buT_23o&t=193s) Um, and, you know, over the course of that time, obviously, you know, when, when we got started, um, in the end of twenty twenty-three, obviously, agents were not even a concept.

[3:20](https://www.youtube.com/watch?v=MI83buT_23o&t=200s) Um, and now everyone has, you know, everyone's talking about coding agents. People are doing more and more and more. Uh, and, and it's very cool to see. Um, and, and each of these has kind of been almost a, a discrete tier for us.

[3:31](https://www.youtube.com/watch?v=MI83buT_23o&t=211s) Um, and so right, right around a year ago when we were doing the last AI Engineer talk actually, um, the, the biggest use case that we really saw that, that was getting broad adoption was what I'll kind of call these repetitive migrations.

[3:43](https://www.youtube.com/watch?v=MI83buT_23o&t=223s) And so I'm talking, like, JavaScript to TypeScript or, like, upgrading your Angular version from this one to that one or going from this Java version to that Java version or something like that.

[3:54](https://www.youtube.com/watch?v=MI83buT_23o&t=234s) Um, and those, those kinds of tasks in particular, what you typically see is you are--

[4:01](https://www.youtube.com/watch?v=MI83buT_23o&t=241s) you, you have some massive code base that you wanna apply this whole migration for. You have to go file by file and do every single one, and usually, the set of steps is pretty clear, right?

[4:10](https://www.youtube.com/watch?v=MI83buT_23o&t=250s) If you go to the Angular we-website or something like that, it'll tell you, "All right. Here's what you have to do, this, this, this, this, this." And, um, you wanna go and execute each of these steps.

[4:19](https://www.youtube.com/watch?v=MI83buT_23o&t=259s) It's not so routine that there, you know, there's no classical deterministic program that solves that. But there's kind of a clear set of steps, and if you can follow those steps very well, then you can do the task.

[4:29](https://www.youtube.com/watch?v=MI83buT_23o&t=269s) And, you know, this was the thing for us because that was all you could really trust agents to do at the time. You know? You could do harder things once in a while, and you could do some really cool stuff occasionally.

[4:40](https://www.youtube.com/watch?v=MI83buT_23o&t=280s) But as far as something that was consistent enough that you could do it over and over and over, um, these kinds of, like, repetitive migrations that you would be doing for, you know, ten thousand files were, you know, in, in, in many ways, the, the, the easiest thing.

[4:52](https://www.youtube.com/watch?v=MI83buT_23o&t=292s) Which was cool actually because it, it, it was also kind of the, the most annoying thing for humans to do. [chuckles] And I think that's generally been the trend where, um, AI has always done these more boilerplate tasks and the more tedious stuff, the more repetitive stuff, and we get to do the, the, the more fun, creative stuff.

[5:08](https://www.youtube.com/watch?v=MI83buT_23o&t=308s) Um, and obviously, as time has gone on, it's, it's taken on more and more of that boilerplate. But for a problem like this one, a lot of what you need to do is you need Devin to be able to go and execute a set of steps super reliably.

[5:22](https://www.youtube.com/watch?v=MI83buT_23o&t=322s) And so a lot of this was, you know, I would say the big capabilities problems to solve was mostly instruction following. And so we built this system called Playbooks where basically you could just outline a very clear set of steps, have it follow each of those step-by-step, and then do exactly what's said.

[5:37](https://www.youtube.com/watch?v=MI83buT_23o&t=337s) Now, if you think about it, obviously, a lot of software engineering does not fall under the category of literally just follow ten steps step-by-step and do exactly what it said.

[5:44](https://www.youtube.com/watch?v=MI83buT_23o&t=344s) But migration does. And it allowed us to go and actually do these, and, and this was kind of, I would say, the first big use case of Devin that really, um, that really came up.

[5:53](https://www.youtube.com/watch?v=MI83buT_23o&t=353s) I think one of the other big systems that got built around that time, which we've since rebuilt many times, is knowledge or memory, right? Which is, you know, if you're doing the same task over and over and over again, then often the human will have feedback on, "Hey, by the way, you have to remember to do X

[6:08](https://www.youtube.com/watch?v=MI83buT_23o&t=368s) thing," or, "You have to, you know-- you, you need to do Y thing every time."

[6:13](https://www.youtube.com/watch?v=MI83buT_23o&t=373s) When you see this, right? Um, and so basically an ability to, to just maintain and understand the learnings from that and use that to improve the agent in every future one.

[6:22](https://www.youtube.com/watch?v=MI83buT_23o&t=382s) And those were kind of the, the big problems of the time. You know, and that was summer of last year. And around end of summer or fall or so, you know, I think the, the, the kind of big thing that started coming up was as these systems got more and more capable, instead of just doing the most

[6:38](https://www.youtube.com/watch?v=MI83buT_23o&t=398s) routine migrations, you could do, you know, these more s- still pretty isolated, but, but, but, but a bit broader of these general kind of bugs or features where you can actually just tell it what you want to do and have you- have it do it, right?

[6:50](https://www.youtube.com/watch?v=MI83buT_23o&t=410s) And so for example, "Hey Devin, in this, uh, repo select dropdown, can you please just list the currently selected ones at the top? Like, having the checkboxes throughout is, it's just doesn't really..."

[7:00](https://www.youtube.com/watch?v=MI83buT_23o&t=420s) And, and Devin will just go and do that, right? And so if you think about it, it's,

[7:04](https://www.youtube.com/watch?v=MI83buT_23o&t=424s) y- you know, it's, it's, it's something like the kind of a level of task that you would give an intern.

[7:09](https://www.youtube.com/watch?v=MI83buT_23o&t=429s) And there are a few particular things that you have to solve for, um, with this. First of all, usually these, these, these changes are pretty isolated and pretty contained, and so it's one, maybe two files that you really have to look at and change to do a task like this.

[7:23](https://www.youtube.com/watch?v=MI83buT_23o&t=443s) But at least you do still need to be able to set up the repo and work with the repo, right? And so you wanna be able to run lint, you wanna be able to run CI, all of these other things, so you know, at, to at least have the basic checks of whether things work.

[7:36](https://www.youtube.com/watch?v=MI83buT_23o&t=456s) One of the big things that we built around then was the ability to really set up your repository, uh, ahead of time and build a snapshot, um, that, that you could start off, that you could reload, that you could roll back and all of these kinds of primitives as well, right?

[7:48](https://www.youtube.com/watch?v=MI83buT_23o&t=468s) So having a, this clean remote VM that could run all these things, it could run your CI, it could run your linter, uh, and, and so on. Um, but that's when we started to really see, I would say, a bit more broader value, right?

[8:00](https://www.youtube.com/watch?v=MI83buT_23o&t=480s) I mean, migrations is one particular thing, and for that particular thing, we were showing a ton of value, and then we started to see where, you know, with these bug fixes or things like that, you would be able to just generally get value from Devin as, as almost like a junior buddy of yours.

[8:14](https://www.youtube.com/watch?v=MI83buT_23o&t=494s) And then in fall, things really moved towards just much broader bugs and requests. And here it's, you know,

[8:23](https://www.youtube.com/watch?v=MI83buT_23o&t=503s) most, most changes, again, you know, where you, you jumping another order of magnitude. Most changes don't just contain themselves to one file, right? Often you have to go and look, see what's going on.

[8:31](https://www.youtube.com/watch?v=MI83buT_23o&t=511s) You have to diagnose things. You have to figure out what's happening. You have to work across files and make the right changes. Often these changes are, you know, hundreds of lines.

[8:39](https://www.youtube.com/watch?v=MI83buT_23o&t=519s) If it's like, "Hey, I've got this bug, let's figure out what's going on, let's solve it," right?

[8:44](https://www.youtube.com/watch?v=MI83buT_23o&t=524s) And you know, there, there are a lot of things here that, that really started to make sense and really started to be important, but, but one in particular I'll just point out was there's a lot of stuff that you can do with not just looking at the code as text, but thinking of it as this whole hierarchy,

[8:58](https://www.youtube.com/watch?v=MI83buT_23o&t=538s) right? So, so understanding call hierarchies, running a language server, uh, is a big deal. You have Git commit history, which you can look at, which informs how, how these different files relate to one another.

[9:09](https://www.youtube.com/watch?v=MI83buT_23o&t=549s) You have, um, um, obviously you have, like, your linter and things like that, but, but you're able to kind of reference things across files. And so, like, one of the big problems here, I think, was, uh, kind of working with the context of it and getting to the point where it could make changes across several files.

[9:25](https://www.youtube.com/watch?v=MI83buT_23o&t=565s) It could be consistent across those changes. It would be able to understand across the codebase. And here was really the point, I would say, where you started to be able to just tag it and have it do an issue and just have it build it for you.

[9:36](https://www.youtube.com/watch?v=MI83buT_23o&t=576s) Um, and so Slack was a, it was, you know, a, a huge part of the workflow then. Um, and, and, and it was just,

[9:42](https://www.youtube.com/watch?v=MI83buT_23o&t=582s) it, it, it, it made sense because it's where you discuss your issues and it's where you set these things up, right? So you would tag Devin in Slack and say, "Hey, by the way, we've got this bug.

[9:49](https://www.youtube.com/watch?v=MI83buT_23o&t=589s) Please take a look." Or, you know, "Could you please go build this thing?" Uh, this is especially fun part for us because this is right around when we went GA, uh, and a lot of that was because it was, it got to the point where you truly could just get set up with Devin and ask it a

[10:03](https://www.youtube.com/watch?v=MI83buT_23o&t=603s) lot of these broad tasks and, and just have it do it. Um, but, but a lot of these, you know, a, a lot of the work that we did was around having Devin have better and better understanding of the codebase, right?

[10:13](https://www.youtube.com/watch?v=MI83buT_23o&t=613s) And if you think about it, you know, from the human lens, it's the same way, where on your first day on the job, for example, being super fresh on the codebase, it's kinda tough to know exactly what you're supposed to do.

[10:22](https://www.youtube.com/watch?v=MI83buT_23o&t=622s) Like, a lot of these details are things that you understand over time or that a, a representation of the codebase that you build over time, right? Um, and Devin had to do the same thing and had to understand, "How do I plan this task out before I solve it?

[10:34](https://www.youtube.com/watch?v=MI83buT_23o&t=634s) How do I understand all the files that need to be changed? How do I go from there and make that diff?"

[10:41](https://www.youtube.com/watch?v=MI83buT_23o&t=641s) And around spring of this year, um, again, every, every gap is, like, two or three months. You know, the, we, we got to an interesting point, which is once you start to get to harder and harder tasks, you as the human don't necessarily know everything that you want done at the time that you're giving the task, right?

[10:59](https://www.youtube.com/watch?v=MI83buT_23o&t=659s) If you're saying, "Hey, you know, I, I'd like to go and, um, improve the architecture of this," or, you know, "This, this function is slow. Like, let's, let's profile it and look into it and see what needs to be done."

[11:09](https://www.youtube.com/watch?v=MI83buT_23o&t=669s) Or, "Hey, like, you know, we really should, should handle this, this error case better, but, like, let's look at all the possibilities and see what we should, you know, what the right logic should be in each of these," right?

[11:19](https://www.youtube.com/watch?v=MI83buT_23o&t=679s) And basically what it meant is that this whole idea of taking a two-line prompt or a three-line prompt or something and then just having that result in a, a Devin task was, was not sufficient, and you wanted to really be able to work with Devin and specify a lot more.

[11:34](https://www.youtube.com/watch?v=MI83buT_23o&t=694s) And around this time, along with this kind of, like, better codebase intelligence, um, we had a few different things that, that, that, that came up, and so we released DeepWiki, for example.

[11:42](https://www.youtube.com/watch?v=MI83buT_23o&t=702s) Um, and the whole idea of DeepWiki was, you know, funnily enough, is Devin had its own internal representation of the codebase, but it turns out that for humans it, it was great to look at that too, to be able to understand what was going on or to be able to ask questions quickly about the codebase.

[11:56](https://www.youtube.com/watch?v=MI83buT_23o&t=716s) Um, closely related to that was with Search, which was the ability to really just ask questions about a codebase and understand, um, some, some piece of this. A- and a lot of the workflow that really started to come up was actually basically this, this more iterative workflow where the first thing that you would do is you would

[12:12](https://www.youtube.com/watch?v=MI83buT_23o&t=732s) ask a few questions. You would understand. You would basically have a more L2 experience where you can go and explore the codebase with your agent.

[12:20](https://www.youtube.com/watch?v=MI83buT_23o&t=740s) Figure out what has to be done in the task, and then set your agent off to go do that. Because for these more complex tasks, you kind of needed that, right?

[12:29](https://www.youtube.com/watch?v=MI83buT_23o&t=749s) Um, and so, so, you know, that was a, I, I would say, kind of like a big paradigm shift for us then is, is understanding, you know, this is what also came along with Devin 2.0, for example, and the in IDE experience where often, yeah, you wanna be able to have points where you closely monitor Devin for

[12:43](https://www.youtube.com/watch?v=MI83buT_23o&t=763s) 10% of the task, 20% of the task, and then have it do, uh, work on its own for the other 80, 90%.

[12:51](https://www.youtube.com/watch?v=MI83buT_23o&t=771s) Um, and then lastly, most recently in June [laughs] which is now, uh, it was kind of, yeah, the really the ability to just truly just kill your backlog and hand it a ton of tasks and have it do all these at once.

[13:02](https://www.youtube.com/watch?v=MI83buT_23o&t=782s) And, you know, if you think about this task, in, in many ways, I would say it's, it's almost like a culmination of, of many of these different things that, that had to be done in the past.

[13:09](https://www.youtube.com/watch?v=MI83buT_23o&t=789s) You have to work with all of these systems, obviously. You have to integrate into all these. Certainly, you wanna be able to, to work with Linear or with Jira or systems like that.

[13:15](https://www.youtube.com/watch?v=MI83buT_23o&t=795s) But- Right ... you have to be able to scope out a task to understand what's meant by what's going on. You have to decide when to go to the human for more approval or for questions or things like that.

[13:25](https://www.youtube.com/watch?v=MI83buT_23o&t=805s) You have to work across several different files. Often you have to understand even what repo is the right repo to make the change in if, if your, if your org has multiple repos, or what part of the code base is the right part of the code base that needs to change.

[13:37](https://www.youtube.com/watch?v=MI83buT_23o&t=817s) Um, and to really get to the point where you can go and do this more autonomously,

[13:42](https://www.youtube.com/watch?v=MI83buT_23o&t=822s) first of all, um, you, you have to have, like, a, a really great sense of confidence, right? And so, um, you know, rather than just going off and doing things immediately, you have to be able to say, "Okay, I am quite sure that this is the task, and I'm gonna go execute it now," versus, "I don't understand

[13:57](https://www.youtube.com/watch?v=MI83buT_23o&t=837s) what's going on. Human, please give me help," [laughs] basically, right? But, but the other piece of it is

[14:04](https://www.youtube.com/watch?v=MI83buT_23o&t=844s) this is, I think, the era where testing and this asynchronous testing gets really, really important, right? Which is if you want something to just deliver entire PRs for you for tasks that you do, especially for these larger tasks, you wanna know that it is ha- can, can test it itself, and often the, the agent actually needs this

[14:20](https://www.youtube.com/watch?v=MI83buT_23o&t=860s) iterative loop to be able to go and do that, right? So it needs to be able to run all the code locally. It needs to know what to test.

[14:26](https://www.youtube.com/watch?v=MI83buT_23o&t=866s) It needs to know what to look for. Um, and in many ways, it's just a, a much higher context problem to solve for, right, is this testing itself.

[14:34](https://www.youtube.com/watch?v=MI83buT_23o&t=874s) And that brings us to now. And obviously it's a, it's a pretty fun time to see because now what we're thinking about is, hey, maybe if instead of it doing it just one task, it's, you know, how, how do we think about tackling an entire project, right?

[14:46](https://www.youtube.com/watch?v=MI83buT_23o&t=886s) And after we do a project, you know, what, what goes after that? A- A- A- And maybe one point that I would just p- make here is

[14:53](https://www.youtube.com/watch?v=MI83buT_23o&t=893s) we talk about all these 2Xs, you know, that happen every couple months. And I think from a kind of cosmic perspective, all the 2Xs look the same, right? But in practice, every 2X actually is a different one, right?

[15:04](https://www.youtube.com/watch?v=MI83buT_23o&t=904s) And so when we were just doing, you know, tab completion, line, single line completion, it really was just a text problem. It is just, like, taking the single file so far and just predict what the line is next, right?

[15:15](https://www.youtube.com/watch?v=MI83buT_23o&t=915s) Over the last year or year and a half, we've had to think about so much more. How do you, how do you work with the human in Linear or Slack or Jira?

[15:22](https://www.youtube.com/watch?v=MI83buT_23o&t=922s) How do you take in feedback or steering? Um, how, how do you help the human plan out and do all these things, right? And moreover, obviously, there's a ton of this, the tooling and the capabilities work that have to be done of how does, how does Devin test on its own?

[15:35](https://www.youtube.com/watch?v=MI83buT_23o&t=935s) How does Devin, um, uh, you know, make a lot of these longer term decisions on its own? How does it debug its own outputs or, or run the right shell commands to figure out what the feedback is, uh, and go from there?

[15:47](https://www.youtube.com/watch?v=MI83buT_23o&t=947s) And so it's super exciting now that there's a lot more, uh, there's a lot more coding agents in the space. It's, uh, it's, it's very fun to see, and I think that, you know, we- we're [laughs] gonna see another 16 to 64 X over the next twelve months as well.

[15:59](https://www.youtube.com/watch?v=MI83buT_23o&t=959s) And, uh, and, and so, yeah, sup- super, super excited.

[16:03](https://www.youtube.com/watch?v=MI83buT_23o&t=963s) Awesome. Well, that's all. Thank you guys so much for having me. [audience applauding] [upbeat music]
