AI Engineer Europe 2026
Your coding agent doesn't always follow your rules
Read the talk
Your coding agent doesn't always follow your rules
A coding agent can announce completion before the work meets your requirements. Talha Sheikh’s Vector walkthrough shows how explicit checks turn that announcement into a feedback loop.
From a talk by Talha Sheikh
The agent finishes. You still have to check.
You give Claude Code a feature to build. It splits the work into subtasks, launches multiple subagents, and works through them until it announces completion. Everything looks promising—until you run the result. Something fails. You ask for a repair, receive another assurance, and discover another small omission.
Talha Sheikh’s desired workflow is more appealing: play Cyberpunk on his Xbox while the agent does the work. Instead, he keeps interrupting to request fixes. A specification, detailed instructions, or almost no instructions still leave him checking the result. The user has become the enforcement layer. The agent says it is done, but nothing else establishes whether that is true.
The missing piece is a deterministic check between reported completion and accepted completion. It must answer more than whether the agent produced something: did it complete the task in the way the user requested? That requires turning at least some expectations into criteria that can actually be checked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn completion into a checked retry loop
Sheikh’s first implementation, Vector V1, uses Claude Code hooks to invoke a checker automatically when the agent finishes. He describes this as finishing a session; current Claude Code terminology distinguishes the Stop event, when the agent finishes responding, from session termination. The relevant mechanism is a hook that can block completion and return a reason, with protection against indefinite continuation.
The configuration defines what Vector should test. The resulting workflow is straightforward:
- The agent attempts to finish its work.
- The hook invokes the configured checks.
- A failed check returns feedback to the agent.
- The agent repairs the work and tries again.
- The checks run again before the result is accepted.
For a JavaScript project with test and typecheck scripts, a check command could combine them like this:
bash
#!/usr/bin/env bash
set -euo pipefail
pnpm test
pnpm run typecheck
The command exits unsuccessfully if either check fails. The hook integration supplies the other half of the mechanism: getting that failure back to the agent instead of letting it disappear into a terminal log.
In the output Sheikh walks through, checks initially fail, the agent retries, and eventually all the checks pass. That progression is the useful demonstration: completion becomes something the surrounding system evaluates, rather than something the agent merely declares. The target is trust in task completion, and Sheikh uses Claude as a stand-in for coding agents generally.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Better models and better instructions leave a verification question
After showing Vector at events, Sheikh encounters an objection. He recounts an unnamed Anthropic engineer predicting that another agent or a sufficiently capable model would make enforcement unnecessary. That raises an uncomfortable possibility for someone building a verification tool: perhaps the next model will erase the problem. He also invokes Project Glasswing and Claude Mythos Preview as examples of optimism about advancing capabilities. Glasswing is a defensive cybersecurity initiative using that model, not a promise that verification will become unnecessary.
Capability and reliability are different questions. A more capable model may solve tasks it previously could not solve. The operational question remains whether a particular result satisfies the requirements. Better performance does not, by itself, supply the check that establishes that result.
The same distinction applies to the inputs surrounding the model. A better specification, useful MCP servers, well-chosen subagents, and the right context are all worth providing. They help the agent understand and perform the work. But instructions describe what should happen; verification examines whether it happened. Improving the former does not automatically implement the latter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Can harness investment reduce model spending?
Once checks constrain the acceptable output, Sheikh proposes that a smaller model such as Haiku, or an open-source model, might become sufficient for some tasks. A frontier model such as Opus is one route to producing the result; a cheaper model inside a stronger verification loop is another possible route.
His proposed tradeoff has three positions:
| Approach | Where the effort goes |
|---|---|
| Frontier model | Greater reliance on model capability |
| Vector with some guardrails | Model capability plus selected checks |
| More extensive guardrails | More time designing the harness, potentially less model spending |
He also mentions asynchronous tasks as another possibility. This is a cost hypothesis, not measured savings: the talk supplies no task-specific benchmark or comparison accounting for check execution and retries. The practical question is whether the extra harness work makes a less expensive model sufficient for the required outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A portable contract, with checks developers define
Further conversations expose a portability problem. Sheikh sees Anthropic, his own company, and Facebook/Meta building their own enforcement systems. A tool tailored to his requirements will not automatically fit someone else’s. Different teams need to enforce different things, so the reusable element must be the pattern rather than a fixed collection of rules.
The proposed pattern is language agnostic and lets developers supply their own enforcement. It can place checks at several points in a workflow:
- Conversation completion: Check the work when the agent attempts to finish.
- Before committing: Check the changes before they enter a commit.
- Multi-agent workflows: Check work within a larger collaboration between agents.
- Asynchronous operations: Check background work when it reaches the relevant checkpoint.
- LLM judgment: Invoke another model to evaluate criteria that call for judgment.
These are proposed integration points for the shared pattern, beyond the initial hook-driven walkthrough.
The last option makes an important distinction visible: deterministic invocation does not require every evaluator to be deterministic. A system can reliably run a particular check at a particular point while that check uses an LLM whose judgment varies. Cross-language portability depends on being able to invoke the checks through a dependable execution mechanism, not on writing every checker in the same language.
The contract connects a task to the conditions required for its fulfillment. Developers define the checks in between. That separates the reusable orchestration—when to verify and how to return feedback—from the project-specific meaning of acceptable work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different systems, a recurring feedback loop
Sheikh connects this pattern to several industry examples, starting with Anthropic’s executor–advisor arrangement. One agent performs the coding work while an advisor provides feedback that can change its course. The current advisor documentation describes strategic guidance rather than deterministic verification; the advisor itself has no tools. The useful comparison is the feedback loop, not a guarantee that advice establishes correctness.
He next points to OpenAI’s harness engineering: the agent receives tools and context within an environment designed to support checking its work. Then comes Qodo’s code review, where review findings and issues feed back into a pull request after the agent has produced changes. In Sheikh’s comparison, these approaches share a refusal to treat generated code as the end of the process, even though their checks and feedback mechanisms differ.
A WorkOS presentation supplies the compact formulation: “Enforce, don’t instruct.” Here, enforcement means making verification part of the workflow itself, rather than relying on the agent to remember that it was asked to check.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Verification sets the pace
This explains why fast code generation does not remove the need to proceed carefully. The agent can produce changes quickly; someone or something still has to establish that they work. Sheikh connects that constraint to calls to go slowly, including a keynote’s blunter instruction to “slow the hell down.” Verification is the work that remains between generating a change and trusting it.
The resulting shift is in where engineering value resides: from the code produced to the verification designed around it. The relevant skill expands from being able to write code to being able to establish that the code satisfies its purpose.
Sheikh’s closing recommendation is to work on the harness and the verification system. The hook, the configured checks, and the failure feedback are what let a result improve without requiring the user to supply every correction manually.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Getting Vector—and paying to build it
Asked whether the tool exists publicly and can be used, Sheikh says yes: it is called Vector Harness, and he offers to share it through a LinkedIn message. An audience member asks him to restore the LinkedIn slide, and he does. The offered route is direct contact; he does not walk through installation or deployment.
The final question playfully returns to the cost argument: how does using smaller models square with allegations that Sheikh is a top token spender at his company? His reply is that he needs those tokens to build the verification layer. The exchange ends on the expense of building the harness, rather than a claimed measurement of what it saves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Hook configuration and lifecycle events, including checks that can block an agent from stopping.
Anthropic's defensive cybersecurity initiative using Claude Mythos Preview.
OpenAI's account of building an agent-oriented engineering environment with context, tests and mechanically enforced constraints.
How Qodo places contextual code-review findings and rule checks into pull requests.
Further reading
Sheikh explains Vector's developer-defined checks and proposed integration points across agent workflows.
WorkOS shares Nick Nisi's presentation and its lessons about enforcement, guidance and measurement.
Updates since the talk
Current documentation for pairing an executor model with an advisor that supplies strategic guidance.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hello, hello.
- 0:16
Um, have you ever given a task to Claude Code? And, uh, you give it a feature, and you be like, "Okay, cool. Uh, can you build this for me?"
- 0:24
And Claude Code starts putting it out into subtasks, and you see it where, okay, this is pretty cool. You see it running on multiple sub-agents. And you're like, "All right, this is really cool."
- 0:33
And you can see, like, ripping through all of your tasks, sub-agents being completed, and it gives you, like, a final output, "Task completed." Like, amazing. Great. But when you actually try to run it, and you'd be like, "Oh, well, it's, it's not.
- 0:45
Something has failed. Hey, Claude, can you fix this a little bit?" thing. Oh, well, let's try it again. Okay, it's fixed. Everything should be working. Oh, no. Actually, just this tiny little thing is just missing.
- 0:54
So... And that's, and that's what my talk is about. All I want to do is play Cyberpunk, uh, on my Xbox while have Claude Code do, do some work for me.
- 1:04
And what I'd realized was the problem is, the problem is that I kept on, uh, telling Claude like, "Hey, fix this, fix that, fix this, fix that," even though if I give it a spec, if I give it some instructions, if I give it little to no instructions, every time there is something that I need to tell
- 1:17
it. So what that means is, like, I am the enforcement. I am the enforcement layer. I have to tell Claude on what exactly you need to do and how exactly this needs to be enforced.
- 1:30
So the agent says it's done, and but you have to check it anyway because there is nothing else that can check it for you.
- 1:38
So what I wanted was something to be very deterministic. So when an agent says it's completed, have this enforcement layer deterministically check something like whether it's actually been done or not.
- 1:48
Actually, the way I wanted it to be done because it says it is done, but is it the way that I want it? So I needed some way, deterministic way to do that.
- 1:57
So I tried it. I built my own Vector. I called it my own product called Vector V1, and it deterministically checks Claude's output. And the way it did that is through using Claude Hooks.
- 2:07
So that way, whenever Claude finishes its, a session, it automatically, the hook calls my Vector product or program, and it checks it for me. Cool, and this is how it essentially looks.
- 2:17
So I basically give it a config file, define all of my testing over here, like what I need it to be checked, and if it fails, it can actually keep on telling Claude like, "Hey, look, this is failing.
- 2:26
Try again, try again, try again." Sorry if it's a little bit
- 2:31
little. So you can see one of the test outputs over here. So it's like, okay, first the test, test passed, first failed, then it retries again, then all of the things pass.
- 2:39
Okay, cool. So what that means is it's not about whether Claude can actually do the task. It's about trust. Can I trust Claude to actually do everything for me?
- 2:48
And by the way, when I say Claude, I'm just talking in general about LLM agents in general. Is when I give a task to a coding agent, does it actually complete it?
- 2:56
So yeah. So and they-- That's something that... And, and what I started doing was s-started telling this about to people about and going to different events, and it was, it was really cool.
- 3:06
Like, "Hey, look, what about this verification feature that I built?" It was so good. It was so amazing. And then I met one of the, uh, Anthropic engineers, and, um, he, and, and they just told me that we're not gonna need this anymore.
- 3:18
Like, we'll have, like, another agent or another model that it will be so smart that you won't need enforcement. Okay.
- 3:27
Um, so crisis mode. Did I just waste my time? What-- Did I just... Like, what was, what was the point of all of this stuff? But let's dig in a little bit deeper.
- 3:37
And then also they released this Project Glasswing that shows Project Mythos, which is supposed to be so good that it will solve everything for us.
- 3:45
So what I-- So when I started thinking about it like, okay, what is it that is actually happening? When a new model comes out, does it, it increases in capability, but that's not necessarily the same thing as reliability.
- 3:56
Sure, the models may become, may become a little more capable, but are they more reliable?
- 4:02
The other thing is, like, another ar-argument is like, "Oh, well, I can have the best spec. I can have the best MCP servers. I can have the best sub-agents.
- 4:09
I can get all the right context to it." Amazing. We should do that. But instr-- giving Claude instructions is not the same thing as giving it verification. So you can give as much instructions as you want, very good instructions, very little instructions, but you still will need to verify.
- 4:24
And what I'd realized was that be-- just-- What I realized that having these small guardrails or having as many guardrails as you want, technically you can use a smaller model like a Haiku or even like an open source models.
- 4:37
Because it's got these guardrails on, it'll most likely be succinct and get you to the output that you want. So in theory, what it means is that if you use a frontier model like an Opus, uh, Opus model that can get you a task, okay, cool, that will be the most expensive one.
- 4:50
You can have Vector with a little bit of guardrails, but it gives you a little bit cheaper. But if you put on more guardrails, that means invest a little bit more time in the harness itself, like you can reduce the cost drastically, or maybe even use like async tasks as well.
- 5:04
So, okay, I was feeling, I was feeling good, and I started talking about, about Vector again at two different events. And as, as I spoke to more and more people, um,
- 5:16
what it-- There is missing, something missing here. What it-- When I spoke to more people, what I'd realized was everybody's building their own stuff. Anthropic is building their own stuff.
- 5:25
My company is building their own stuff about enforcement. Facebook is building their own stuff. Meta is-- Ev-every company is building their own thing. So if I have built, if I build something that is specific to me, then I can't really share it with others because everybody has their own way of doing it.
- 5:40
And what I enforce ne-ne-doesn't necessarily mean that somebody else would enforce the same thing. So what that meant was, what I realized was, okay, so it's actually a pattern.
- 5:50
So it has to be a pattern that is applicable to everyone. So what that means is that we can-- it has to be language agnostic. It has to be something that, uh, can be shared by everybody else, and everybody can bring their own version of enforcement to it.
- 6:03
And that's-- And, and it should run on every level. So it should start, start off with in conversation when a conversation ends. You can have checks when, uh, before committing.
- 6:12
You can have checks when you're pa- as part of a multi-agent workflow. You can have it, uh, on checks on asynchronous operation or asynchronous agents. And as well as you can have a check that non-deterministically calls like, um, LLM and LLM-as-a-judge sort of a thing.
- 6:26
And it can run on any, any different language on any different code. As long as there's a capability to run it deterministically, we sh- we can have that. So what, what I realized was, what we needed was essentially a contract that just says like, "Hey, given this task, I want you to fulfill this."
- 6:42
What is in the middle that you can-- the developers themselves can define.
- 6:47
So this idea is really cool, and it was like, okay, so we're moving towards w- we want verification always. All right, cool. Um, so a lot of different companies have actually started doing this as well.
- 6:58
So Claude Co-- uh, Anthropic has r- recently released their new thing called Executor Advisor pattern, where you've got one agent that actually does all the code, all the code work, and then there's advisor that, you know, feeds in...
- 7:09
essentially creates a feedback loop, or in other words, verify. Anthropic, uh, sorry, uh, OpenAI build their own Harness engineering, and it's the same idea. Like, you give an agent a lot of, a lot of things to do, but how do you verify it to work?
- 7:22
You give it different tools. You give it different contexts, and that's essentially what a harness is for OpenAI. There are companies like Kudo, who are, who are over here, that provide a very comprehensive code reviews.
- 7:33
And again, it's the same thing. The, the agent has done all of its work, but do you trust it? No. So what do we do? You do a very me- comprehensive PR review with all the different issues and findings and create this feedback loop.
- 7:46
Some, um, something from today as well from WorkOS. So it says, "Enforce, don't instruct." So it is all about, like, running these checks deterministically. When I say checks, it's just about the verification.
- 8:00
Another one, which is my favorite, is, uh, one of the favorites, like you still have to go slow. And the reason for that is not because the cl- the agent themselves are not able to produce code as fast as they want, but it's because the verification layer.
- 8:11
You need to verify that everything is working or not.
- 8:14
And my favorite [laughs] is, is this one in, in our keynote, is to slow the, slow the hell down. [laughs]
- 8:22
So, so what is the shift that we're seeing here?
- 8:25
Well, initially what we thought was, like, the value is in the code that we create. But it's actually now, in reality, is, uh, what we're seeing here is the verification that we design.
- 8:35
So it's not about can you code, but can you verify?
- 8:41
So TLDR is, uh, work on the harness and not on the code. So you work on the verification system, and that produces a little bit better, better in outputs.
- 8:51
And that's it. Thank you. [clapping] Any questions? I've got 40 seconds. [laughs]
- 9:07
Yep.
- 9:08
Does it exist? Is it public? Can we use it?
- 9:10
Yeah. [laughs] Yeah, yeah, yes, it is public. Um, it's called Vector Harness, but if you, if you send me a message on LinkedIn, I can share that with you.
- 9:17
Can you put the LinkedIn back up?
- 9:19
Oh, there you go.
- 9:20
Thanks.
- 9:25
Cool. Yeah.
- 9:29
You mentioned that adding the verification layer allows you to use smaller models. Uh, what do you say to the allegations that you're a top token spender at your company? [laughs] [laughs]
- 9:40
I need, uh, I need those tokens to build a verification layer. [laughs]
- 9:46
Cool. Um, I think that's it.
- 9:53
Whoo. [clapping] [outro jingle]