AI Engineer Europe 2026
How Lovable self-improves every hour
Read the talk
How Lovable learns from stuck agents
Lovable turns recovered sessions into reusable guidance and agent complaints into reviewed fixes, using production feedback to decide what to keep, change or remove.
From a talk by Benjamin Verbeek
Before you start: Familiarity with AI agents, tool calls and basic software development will help; the TypeScript example requires only an understanding of strings and replacement.
Why explain the same mistake twice?
Why does an agent need the same mistake explained over and over again? A correction helps the current conversation, but the next attempt—or the next user—can encounter the same failure. Continuous learning requires turning that correction into a change that survives the session. Benjamin Verbeek, a member of technical staff at Lovable with a background in satellites, particle physics and fusion reactors, introduces two approaches toward that goal: learning better ways to use the existing product, and repairing the product itself.
Lovable’s interface puts a chat beside a sandbox preview. Describe the software you want, inspect the result, test it and ship it. Verbeek tentatively credits Lovable with coining “vibe coding,” although Lovable’s own introduction to the term credits Andrej Karpathy. The relevant product idea is software creation without having to inspect the code between an instruction and its visible result.
That interface produces an unusually useful learning environment. Users can spend a long time in one conversation, working toward an artifact they care about shipping. Compared with short, unrelated chat sessions, this gives the agent more context about both the project and the person building it.
Verbeek frames the audience as the 99 percent who cannot code: an expression of Lovable’s ambition to make software creation broadly accessible, rather than a measured population estimate. He compares these users to startups adopting new technology. They pursue an outcome without being constrained by older implementation habits, and keep trying until it works. The product has to make that persistence productive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A technical blocker can end the project
When Verbeek joined roughly a year before the talk, Lovable had a few thousand users. Verbeek reports that Lovable was creating more than 200,000 projects per day at the time of the talk. That is project creation, not a count of published websites; his accompanying claim about Lovable’s share of new websites comes without a denominator. Scale creates both a large supply of learning examples and operational pressure. He recalls GitHub banning Lovable on his first day because it created too many repositories, and says the company overwhelmed several cloud providers along the way.
For a technical user, AI-assisted development can alternate between rapid progress and recoverable friction. Verbeek illustrates the good stretches as tenfold or hundredfold acceleration, not as a measured benchmark. A yellow friction point might yield to a more precise prompt. A red blocker might require editing configuration, changing a setting, supplying an environment variable or adding an API key. A developer can intervene and resume.
A nontechnical user may prompt past the yellow section but leave at the first red one. They can abandon the project before experiencing a successful end-to-end result. The important reliability target is whether the user can finish, not just whether the agent usually makes progress. As models improve, more of these users’ requirements become achievable; Lovable’s focus over the preceding year was removing the blockers that still prevent them from getting there.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Detect the failure, then choose the right intervention
A session can reveal that a user is stuck without requiring them to diagnose the cause. Repeated requests suggest that the first attempt did not work. Complaints about an implementation, explicit failure reports and apparent premature abandonment provide other signals. An LLM judge can inspect sessions for these patterns and flag likely failures. A first request to improve something is not, by itself, evidence that the user is stuck.
The next question is whether the current product can solve the task at all.
| Failure class | What is missing? | Appropriate response |
|---|---|---|
| Solvable today | The right prompting or approach | Reuse successful guidance |
| Unsupported, but straightforward | A bug fix or small capability | Improve the product |
| Unsupported and difficult | Substantial engineering | Longer development work |
The first class includes tasks that persistent users can already complete. The second includes things that should work but do not. The third may require weeks of AI-assisted engineering. Verbeek’s immediate targets are the first two: if a task is solvable, it should work for more than the users willing to struggle; if a fix is straightforward, the team should ship it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a successful recovery into reusable knowledge
For tasks the product can already solve, Lovable builds what Verbeek calls a Stack Overflow for Lovable: a collection of problems and solutions that the agent can use. Consider a user complaining that a website becomes laggy while scrolling. Initially, that is simply a new task. The agent attempts a performance fix by quantizing animations, but the user reports that the site is now jumpy as well as laggy. The repeated request and complaint about the attempted fix make this a clearer stuck session.
The conversation may continue through several attempts. Some users leave; others persist long enough for the agent to find the cause. In this example, individual gradients on overlay text were making the animation slow. Once the agent reaches a successful solution, the session’s isStuck state returns to false. The useful event is the transition from stuck to resolved, with care to distinguish resolution from a user simply giving up.
That recovery supplies a concrete extraction question: What information could have been injected at the initial request to reach the successful solution directly? The goal is to spare the next user the failed intermediate attempts, rather than merely archive the conversation.
Lovable then clusters similar issues to identify the reusable information. Without this step, the knowledge bank could fill with near-duplicate entries that effectively say to perform one exact action for one exact prompt. Clustering aims to capture the common problem and solution without overfitting to an individual session.
In the workflow described in the talk, an external reviewer—usually another agent, sometimes a human when there is uncertainty—generates and runs a quick evaluation against the examples in the cluster. The question is whether the proposed guidance resolves those cases. Accepted entries become part of a continually updated problem-and-solution bank.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Useful context has to keep earning its place
While the main agent works, a lightweight model watches for issues that have an answer in the knowledge bank. When it finds a relevant answer, it injects that context into the main agent. For a small sample of eligible cases, however, the system sends a blank instead. These projects could have received the guidance but did not, providing a comparison group for the projects that received it.
The production loop asks whether the guidance helps the whole project succeed:
- Identify projects eligible for a particular knowledge injection.
- Compare projects that received it with eligible projects that received nothing.
- Increase use of guidance associated with better project outcomes.
- Reduce use of guidance associated with worse outcomes.
The evaluation target is overall project success, rather than whether the injected text sounds helpful. The talk does not specify the holdout allocation method or experiment size.
Knowledge retirement is part of learning. A new model may no longer need advice that helped its predecessor. A product change can make an old workaround obsolete. Lovable therefore rebalances usage and discards entries as the surrounding system changes. Keeping every previously useful answer would accumulate stale context that can interfere with the agent’s work.
Verbeek reports fewer fix-intent or stuck messages and more deployments in early production data, without supplying numerical effect sizes in the talk. He says subsequent results were better. Deployment matters because it indicates that a user completed the project instead of encountering a blocker severe enough to abandon it.
The collected problems also support internal model rankings. Verbeek says all models at the top of Lovable’s internal ranking used the Stack Overflow information. Some entries were undisclosed, so the result describes Lovable’s internal problem set and evaluation rather than a reproducible public leaderboard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the working agent a way to report missing capabilities
A knowledge entry cannot fix a capability the product does not have. For bugs and unsupported tasks that should be straightforward, Lovable uses a second feedback loop. The analogy is an employee who cannot complete an assignment with the available tools: they would complain to a manager or explain the problem in Slack. Lovable gives its agent a similar outlet to report problems directly to its creators.
An external reviewer asked what could be improved after every iteration is under pressure to produce an answer even when the iteration went well. Verbeek argues that this creates noise and encourages overfitting to it. Instead, the working agent is prompted to report only substantial frustration. That threshold can be tuned to increase the proportion of useful reports.
The vent/send feedback tool asks the agent to report tooling, documentation or platform behavior that materially slows or degrades its work. Its triggers include:
- Missing capabilities: absent or unsuitable tools.
- Unclear contracts: confusing tool names, unexpected parameters or mismatched schemas.
- Conflicting guidance: documentation or instructions that disagree.
- Platform failures: broken behavior or repeated attempts blocked by environment limitations.
These are problems the team hopes are absent, but can act on if the agent identifies them.
The working agent also has diagnostic context that the user may lack. It has already attempted the task, observed the tool behavior and perhaps spent several turns investigating. Asking it to report the obstacle preserves that context at the point where the problem is encountered.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make workflow friction legible to engineers
One complaint sent directly to Slack concerned Framer Motion’s TypeScript types. The agent wanted to provide four numbers for what Verbeek thought was a cubic Bézier curve, and objected to the casting required by its attempted implementation. Verbeek questions whether that particular complaint was relevant, while suggesting that the use case could have been simplified. This is a specific historical interaction, not evidence that four-number curves are generally unsupported: current Motion documentation supports them, and the talk does not establish the library version or exact type error.
The complaint’s conversational form is useful in its own right. Engineers recognize the experience of fighting a tool to express something simple. That makes the report easy to interpret and gives them a starting point for deciding whether to simplify the workflow, clarify the interface or dismiss an irrelevant complaint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeated complaints expose an incomplete repair
A more concrete report concerned a copy tool that could not move certain uploaded files into another document store. Engineers checked the tool and found that it worked, but the agent identified a condition their checks had missed: filenames containing raw spaces failed. Verbeek recalls roughly twenty complaints about the copy failure in the first hour after launching the feedback tool.
The first repair replaced spaces with underscores. Reports continued. The team then identified non-breaking spaces in screenshot filenames associated with WhatsApp or Mac, which their regular expression had not replaced. The distinction can be illustrated in TypeScript with two filenames that look almost identical:
typescript
const filenames = [
"Screenshot 1.png",
"Screenshot\u00A01.png",
];
const replaceOrdinarySpaces = (name: string): string =>
name.replace(/ /g, "_");
const results = filenames.map((name) => ({
before: JSON.stringify(name),
after: JSON.stringify(replaceOrdinarySpaces(name)),
containsNonBreakingSpace: name.includes("\u00A0"),
}));
console.table(results);
The ordinary space becomes _; the example’s U+00A0 remains unchanged. These teaching filenames illustrate why matching one space character is narrower than handling the filenames users actually supply. The talk does not identify the original regex or exact Unicode code point.
Further reports identified other special characters, and the team continued repairing the tool until, Verbeek says, the issue stopped occurring. He does not supply the final implementation or a follow-up duration. The useful mechanism is the repeated feedback: each incomplete repair left a remaining class of failures that the agent could describe.
Tool-failure telemetry could show that something failed. The complaints explained the circumstances of failure closely enough for engineers to act. That is the additional value here: not an absence of logs, but a report that connects a symptom to a likely cause.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a Slack report to a reviewed production fix
The initial route was direct: the agent encountered an issue, called the vent tool and sent a report to Slack. Verbeek started with a restricted channel because he was unsure whether the volume would become spam. Lovable’s head of product was enthusiastic enough to read every message. As the workflow developed, a monitoring agent took on deduplication, investigation and pull-request creation. Developers still reviewed the changes and, in many cases, merged them into production.
The stream also became an incident signal. When Verbeek asks what explains spikes in vent counts, the exchange identifies outages and other platform incidents. Broken sandboxes are one example: many agents encounter the same failing capability and complain at once. The reports can therefore indicate both that an incident is happening and which functionality is affected.
There is an economic advantage to collecting feedback inline. The working agent has already processed the context with a strong model. Asking it to identify a material obstacle uses that existing understanding, whereas an equally capable external reviewer would need to read a large context again. Verbeek describes inline feedback as much cheaper comparatively, without providing a measured cost comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The remaining loop: review and evaluate the fix
The agent eventually supplied feedback about the feedback tool itself: it was too easy to send a report, and there was no way to retract it. Verbeek jokingly describes the agent as embarrassed by what it had sent to Slack. Underneath the anthropomorphism is another tool-design problem: the reporting interface made an action easy without giving the agent a way to correct it afterward.
By the end of the talk, Verbeek is receiving review requests on his phone for pull requests produced by the automation. He inspects them before they can be merged. Automated investigation and patch creation do not yet mean autonomous production approval.
The unfinished work is to connect detection, merged fixes and continuing review and evaluation into a complete improvement loop. Finding a shortcoming and proposing a patch are only part of that system; the resulting change must also be assessed as the product continues to evolve. Verbeek closes by inviting others to help build that loop toward fully automated continual improvement, with human review still part of the workflow he has demonstrated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Examples of named easing functions, four-number Bézier curves and easing modifiers.
Lovable’s explanation of building through natural-language instructions, including its attribution of the term to Andrej Karpathy.
Updates since the talk
Verbeek’s companion account of knowledge injection, agent complaints, production evaluation and human-reviewed fixes.
Read the complete timestamped transcript
- 0:00
[upbeat music] All right.
- 0:17
Let's get this party started. Do we have any Lovable users in here?
- 0:23
Good, good. A few. Very nice. Uh, warm welcome to, to this talk on how Lovable is improving every hour, including this hour as I'm speaking. Uh, I'm Benjamin Verbeek.
- 0:34
I'm a member of technical staff of Lovable. It's so amazing that we are filling up this bonus room. I think we're around three times as many people as would have fit in the original, so thanks to the organizers.
- 0:44
Um, this is my background. I used to work with satellites, particle physics, and fusion reactors, so I have a physics background. And today, I'm working at Lovable and working towards what is maybe the holy grail of AI engineering right now, which is continuous learning at scale.
- 1:03
How do we learn from mistakes? So we've probably all experienced working with an agent and having the feeling, "Why do I have to explain the same thing over and over again?"
- 1:14
This is what we want to avoid. We want to have a mistake happen once and then never again. So we need to learn from that. And in this talk, I want to share two ways how we are working towards this goal at Lovable.
- 1:28
This is a Lovable interface. Uh, we were actually one of the first-- uh, I think we coined the term vibe coding, so coding without actually looking at code. So you have a chat interface where you describe what you want to happen, and you have a modal sandbox, for example, where you can see directly what you- you're creating.
- 1:49
And I think in many ways, this is the way we should be building software. The code was always just an annoying technical layer in between to create what we wanted.
- 1:59
But ideally, you just say what you want, you see it, you test it, and you ship it.
- 2:05
We have some nice benefits with the Lovable platform, which is that people stick to one project for quite a long time. This chat can go on forever, essentially, and people have, like, this one artifact that they are really passionate about and they want to ship.
- 2:20
So we can learn a lot about this one thing versus, for example, a, a chat agent where you have very, very short conversations. We can actually get to know the user in quite detail.
- 2:36
Another exciting thing is that we're building for the ninety-nine percent who can't code. So probably not the people in this room, actually. There are still quite a lot of people, evidently, who use our product, but we are trying to unlock software creation for everyone.
- 2:48
In the future, you will not have to look at code to be able to create software. I think that is the world we all wanna move towards, and we are trying to do that right now.
- 2:56
And in many ways, this is like, uh, building for the future, I think. So a reason that startups succeed with new technology is that they are a bit naive, and they don't really think about all the issues that can happen.
- 3:11
They don't think too much about all the details and implementation. They just go until it works. This is how I like to see our users. They just go until it works, and they're not sort of held back by old paradigms.
- 3:24
This is a wonderful group of people to, to build for, um, and it's scaled quite rapidly. Uh, I joined Lovable around a year ago. At the time, we had a few thousand users.
- 3:36
Now we're creating over two hundred thousand projects per day. That's a significant percentage of all internet websites being created on Lovable. And that scaling journey has been incredibly fun.
- 3:46
We can do so many cool things now, like the things I will talk about in this talk. Uh, it's also very hard. My first day, GitHub banned us because we were creating too many repos.
- 3:55
And we've taken down a number of cloud providers along the way.
- 4:00
How do we succeed with AI? Um, I like to think about it in this way. Technical personas generally have this amazing moment with AI where they are just building and building.
- 4:09
You're accelerating ten times, a hundred times faster than you used to. But sometimes we hit some friction points, this yellow part, and you have to intervene or, like, prompt a bit harder.
- 4:18
And sometimes you even get stuck. Maybe you have to manually go in and change a config somewhere. You need to really do a lot of manual work, change a setting, add an env var, API key, et cetera.
- 4:29
And then you move on, and you really get the good parts of AI, and you can work past the bad parts. You're still annoyed by it, but you can work past it.
- 4:38
A non-technical persona is not really like this. They might prompt their way past the first friction, but as soon as this technical block happens, they generally walk away, and they give up, and they've actually still never experienced successful AI.
- 4:53
This is what we need to minimize. We can never get stuck to a point where a non-technical user cannot get pa-- ca-get past it. And this is a very exciting problem to work on.
- 5:04
Luckily, uh, models are getting better, and the requirements of these people is, uh, generally to a point where we can achieve them. But for the past year, we have been working on making sure none of these red bars happen.
- 5:17
People can never get stuck. How do we do that?
- 5:22
First of all, we want to define, like, what does it mean to be stuck? We probably want to find those cases and then learn from them. So one clear signal of someone being stuck is that they ask for the same thing more than once.
- 5:35
Probably stuck or it didn't work as smoothly as they could. Maybe they're complaining about how something was implemented or that it failed very explicitly, or they gave up on a session that they would otherwise have continued.
- 5:46
And we can have a LLM judge that is just looking at sessions and trying to look for these situations, and they can flag it and say, "Hey, I think this user is stuck."
- 5:57
I want to split being stuck into two different ways. Uh, you can be stuck in a way where it's possible to solve. This is maybe the yellow part that I showed.
- 6:04
And you can get past it with the right prompting. Some users are more willing to go-- put in that effort and get past that point. Uh, others will give up.
- 6:12
But it is possible to solve with the current structure of our product. And then there are tasks that are not solvable, even if you say all the right things.
- 6:21
Maybe we're just not supporting it for some reason.
- 6:24
Um, and there, there are two classes. There's the things that are just done that we don't support. Maybe it's a bug, maybe it's a very, very simple thing we could add to the product.
- 6:33
Uh, and then there are the actual hard things that would take, like, weeks of engineering, AI-assisted engineering effort.
- 6:42
These first two things we should be able to improve on very rapidly, I think. There's no excuse to not improve on these things. If it is solvable, then it should work for everyone.
- 6:50
And if it is just easy to do, we should just ship it. So how do we do that?
- 6:56
Um, some of you might have heard of Stack Overflow, this very, very ancient thing. We had an idea to build a Stack Overflow for Lovable. So we basically learn from-- whenever someone is stuck and has an issue, we try to figure out a solution and give that to the agent.
- 7:15
It could look something like this. Um, I'm complaining to the agent about my website being laggy when I scroll. It's bad performance. Uh, maybe I'm not stuck yet. This is just me complaining about something and giving it a new task.
- 7:28
The agent probably replies something like, "I fixed it. Maybe I quantised animations to make it more performant."
- 7:35
And it was like, "The agent has failed, uh, and it actually made things worse. Now the website is jumpy and laggy. Terrible." Okay, this is a clear case of someone being stuck.
- 7:45
They ask the same thing again. They probably want it-- want the agent to try again, and they're complaining about an implementation that didn't do what they wanted.
- 7:55
This might repeat for a while, where they keep iterating several times. Some people give up at this stage, but at least some people will continue.
- 8:06
And eventually, the agent might find the true solution, which in this case happened to be that the overlay text had individual gradients, which made the animation super low performing.
- 8:16
And now we see that isStuck changes back to false. It succeeded. So this we can flag. We can notice whenever it went from being stuck to not being stuck, and ideally not because the, the user gave up.
- 8:29
This we can flag, and now we have a high signal sample of a problem that was high friction or not solved that was solved. So we have gotten the solution right here.
- 8:40
And the question we just ask is, what should we have injected at the start of this query to jump straight into the solution so the next user does not experience this friction?
- 8:50
What we do is we create a new Lovable Stack Overflow knowledge entry. And in general, we actually do some clustering in-- at this step, where we look for similar issues, we look for what is the actual information we should give, so that it's not over fitting to the specific issue.
- 9:06
We don't want like a million, uh, overflow pages, Stack Overflow pages that are all talking about, "If you get this exact prompt, then you should do this exact thing."
- 9:15
It's not very helpful. We do some clustering,
- 9:18
and then we have an external reviewer, uh, generally an agent, and then maybe in some cases, we have a human if we are uncertain. But in most cases, it's actually just an agent that generates and runs a quick eval on this and sees if this-- did this resolve, uh, the specific examples that we had in this set.
- 9:37
From that, we get a full bank of Lovable Stack Overflow problems and solutions that's being continually updated.
- 9:46
Whenever the model is working, we have a lightweight model that tries to inject that context when needed. If it detects there is an issue and there is an answer, it injects that context into the main agent.
- 9:57
And sometimes it detects that I should inject this, but we inject a blank, so we don't actually send anything for a small sample of use cases.
- 10:05
And this allows us to, with very high signal review, whether this, uh, solution was actually useful in production. And we rate it. We compare the, the group of projects where it was injected and where we-- it, it could have been injected, but it wasn't.
- 10:19
And we say, which of these projects were actually more successful overall? If it was more successful, then we show it more, and if it was less successful, we should show it less.
- 10:27
And this loop is incredibly important. I cannot emphasize enough how important this step is because things are moving around this set of knowledge all the time. It gets stale whenever a new model is released.
- 10:38
It gets stale whenever we change features. It gets stale incredibly quickly. So very often we have to rebalance this, but we also have to throw away a lot of this context.
- 10:49
And this allows us to really be at the frontier of what is solvable right now, but not have a lot of old deprecated knowledge that is giving context rot, um, and also in many cases just hampering the actual, uh, agent.
- 11:05
And this is working at scale. Uh, this is very, very early data, actually. We're, we're doing quite a lot better now. Um, but the number of messages with fix intent or people being stuck is dropping significantly, and we have also a significant number of people that deploy more.
- 11:19
This is one of our key metrics that they actually finish a project. This is a very, very strong signal that it has worked all the way through. They've never gotten stuck so bad that they give it, given up and abandon their project completely.
- 11:34
Um, we also do internal ranking with this, so it's really interesting to see how models perform on this set of problems that we have collected. And with the Stack Overflow information, uh, all models in the top of our ranking use this information.
- 11:50
There's a few hidden entries here, which I unfortunately cannot talk about. Um, but it really, really makes a significant boost in our internal ratings.
- 12:00
And then there's a second set of, uh, being stuck, which is, uh, when it's not actually solvable with the current setup. Maybe there's a bug in our product, um- But in these cases, we think it should be easy in principle.
- 12:15
And what happens if you think something should be possible, but it just doesn't work? You feel significant frustration.
- 12:24
And what would humans do if this happened? Well, if you were given a task and you just don't have the tools to do it, you would probably complain to your boss or go venting in Slack.
- 12:33
So we thought, why not do exactly that, but for the agent? [laughs]
- 12:38
So we basically are asking Lovable, "How are you doing? Can we help you in any way?" And we gave it an outlet to let out its frustrations.
- 12:47
So we asked ourselves, what if we could, uh, let the Lovable agent give direct feedback to its creators? This sounds very scary, I have to say, and it sounds like an absolute insane idea.
- 12:56
Um, what's even more crazy is that it's kind of working.
- 13:01
This kind of detecting issues is something I've heard during this conference, but also a few other times, where you might have an external reviewer looking at the conversation and asking, "What could have been done better to reduce friction?"
- 13:12
One problem with that is that you get a pretty low signal-to-noise ratio because you're basically forcing an answer on every iteration, and the reality is that most iterations actually just work pretty well, and you will overfit to noise.
- 13:27
In this case, it's prompted to only send feedback if it is really frustrated, and you can tune that balance until you get a lot of signal.
- 13:36
So we gave our agent a vent tool, a way to complain to its creators. Um, it looks something like this. It's a vent/send feedback tool. You should use this when tooling docs or platform behavior materially slows or degrades your work, for example, missing or unsuitable tools, unclear tool names, parameters or schemas that are not matching what you
- 13:57
were expecting, confusing or conflicting docs or instructions, broken or unexpected platform behavior, repeated failed attempts at causing, uh, caused by environment limitations. So basically, a list of all the things we think we sh- we should be able to solve and that we hope is not an issue, but if it is, please tell us.
- 14:16
And our users generally don't know what the cause of a problem is. When I'm using Lovable and I get stuck, I generally don't know, but the agent actually has a lot more context.
- 14:25
It has literally been working on the issue, sometimes several turns, and it generally has a lot of context on this problem.
- 14:34
So it can look something like this. [laughs] It actually gets sent directly to our Slack, and the agent says something like, "I'm so annoyed at Framer Motion's TypeScript types for, uh, generating," I think this is a cubic Bézier.
- 14:48
"I just wanna be able to send a list of four numbers. That should be fine. That's all I'm sending. I don't need all this casting gymnastics." Now we can ask ourselves, is this a relevant vent or not?
- 14:59
Um, but in this case, it, it probably just was not relevant for, uh, its use case, and we could have simplified things a lot.
- 15:07
Another benefit of this setup is that it's very easy to understand as a human. It's very relatable that you're complaining about your workflow, and that means that engineers sort of already have all this implicit context on how to alleviate the issue.
- 15:22
Another problem that got reported by the agent was, uh, our copy tool was struggling with certain file names. It couldn't copy it to, uh, another place where we store, store documents, and we were so confused by this.
- 15:36
Like, we checked the tool. It's working. Um, we didn't even know that this tool was failing so regularly until the agent told us. And it said, whenever there's a space, a raw space in the file name, it's failing to copy,
- 15:49
and we got like twenty complaints about this in the first hour of launching this tool, which is crazy. It turns out that, uh, there was an error where it could not copy files with a space in their name.
- 16:02
We fixed it, and we told it, whenever there's a space, just replace it with an underscore, and we kept getting the reports. And then we realized that when you screenshot something in WhatsApp or on Mac, it en- it enters a non-breaking space, which we did not replace in our regex.
- 16:17
And it kept, kept complaining for various other special characters until we did solve it properly, and now this issue never happens again.
- 16:25
I think this is an, a prime example of something that's pretty hard to detect in other ways. You can see the tool failure, but this is just a such a clear case of how to fix it, and we just do it.
- 16:35
I will skip this one. Um, it's essentially this is the flow. The agent experienced an issue. It uses the vent tool. It's sent straight to Slack.
- 16:44
At first, I was very uncertain if this would work, and I didn't wanna spam, spam us. It was like a closed-down channel. It didn't invite so many people. Our head of product was very excited to see this.
- 16:54
Uh, he was reading every single message. Now it's a bit more balanced, and we actually have an agent that's monitoring, removing duplicates, and investigating and creating a PR for all these issues all the time.
- 17:07
And we're still at the point where devs are reviewing and then, in many cases, actually merging this to prod.
- 17:14
This is the number of vents over time. Uh, do we have any guesses what these spikes are? Why do we see spikes in the number of vents caused per time?
- 17:23
Server went down. Server went down. It's an incident, yes. Something broke in the platform. At some point, our sandboxes broke. At some point, something else broke, and the agent is very upset about this. [laughs]
- 17:34
It's complaining a lot. So it turned out that this was actually a prime place to notice when our product was having an incident, and it actually gave a pretty good sense for what the problem was.
- 17:44
It was complaining about the right things in general.
- 17:47
Um, very brief. I'll keep this brief. Um, you get the strong model intelligence versus an external reviewer. You generally don't want top frontier-level intelligence to look through a lot of context, but if it's in line, it's very, very cheap comparatively.
- 18:04
Um, this was an example where the agent was actually giving meta feedback on the venting tool. It said, "It's too easy to send feedback, and I can't pull it back."
- 18:11
It was being ashamed of what it had sent to Slack.
- 18:14
Um, and it's actually the case now that I just get, uh, review requests on my phone from this automation to, "Hey, review this PR that was completely automated." I look through it quickly, and we can merge it.
- 18:27
And we're continuing to work to close this loop of detecting a shortcoming, merging a fix, and then continuously review and eval that. And if you think this is exciting, uh, you should join us and help us close this loop and fully automate, uh, continual improvements.
- 18:43
Thanks for listening. [audience applauds] [upbeat music]