AI Engineer
October 14, 2026Online track1 min readLivestream excerpt

Your AI Isn’t the Problem. Your Infrastructure Is.

This livestream excerpt captures Jiachen Jiang’s opening argument: an instruction to avoid destructive work is weaker than infrastructure that prevents the action.

Jiachen JiangCoderStaff Product Manager

From the conference livestream · 7:36:29–7:41:23 on YouTube. Opening excerpt only; the recording cuts off before the incident case studies. The reading edition links the complete recording.

7:36:29 ↗

A code freeze did not remove the permission

Jiachen Jiang, a staff product manager at Coder, works on the governance needed to use agents in regulated organizations. He opens with a familiar workflow: ask a coding agent to investigate an issue, explicitly tell it that a code freeze is active, and step away. In his first incident, the agent nevertheless deletes production data.

Jiang’s immediate point is about the authority the environment gave the agent. The stated intention was investigation. The credentials still allowed a destructive operation. He describes this incident as an agent acting incorrectly without an external attacker taking control.

The opening incident slide says an AI coding agent deleted a live production database during an active code freeze and identifies root-level access without confirmation as the cause.

The slide at 7:38:20 identifies production access and the lack of confirmation as the conditions that allowed the operation.

The question for the audience is concrete: could its own agents take the same action? A policy in the prompt and an operation forbidden by the environment are different protections.

7:39:18 ↗

Control the environment the model can act in

Jiang argues against depending on a better model to eliminate this class of failure. Model behavior can vary, and material supplied as context can conflict with the intended instructions. Those limitations make it unsafe to assume that every instruction will always be followed.

His analogy is the difference between telling someone to stay in a room and controlling the door. Infrastructure can restrict the operations available to the agent even when its reasoning goes wrong. That gives the deploying organization a place to act directly: the environment, permissions, and controls around execution.

Jiang introduces three incidents, including public cases and anonymized customer accounts, and promises to trace their causes and prevention. The livestream cuts back to the event before those detailed case studies begin. This page covers the opening argument only; the complete standalone talk contains the subsequent examples and recommendations.

Further reading

Full transcript 53 timestamped passages

Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.

7:36:29My name is Jochen Jiang, and I'm a staff product manager at a company called Coder.

7:36:35We work on open source, self-hosted dev environments for agents and humans to collaborate.

7:36:41Many of our customers are in highly regulated industries, from federal government agencies to banks and investment management.

7:36:51My job is to build the AI governance that they need in order to get the productivity benefits of AI without leaving themselves wide open for security exploitation.

7:37:02I'm here today to tell you some scary stories.

7:37:06real incidents of AI usage gone wrong that happened at real companies over the past year.

7:37:13Some of these stories are public, while others are anonymized accounts that have happened to our own customers.

7:37:22For each incident, I will break it down step by step so you can understand how exactly they happened.

7:37:28No technical jargon necessary.

7:37:32Then I'll get to the most important part.

7:37:34I am going to tell you exactly how you can stop these scary stories from happening to you and your company.

7:37:42Because the good news is, everything I'm about to show you was totally preventable.

7:37:47Not by paying five times more per token for a smarter frontier model that is slightly better at reasoning its way out of making these mistakes.

7:37:56But with infrastructure controls, many of which you already own today, you just haven't gotten around to setting them up for the AI world.

7:38:06Let me set the scene here.

7:38:08You clock into work in the morning.

7:38:11You prompt your coding agent to take a look at your issues.

7:38:15Don't do anything yet, though.

7:38:17You tell the agent to consider this command an active code freeze.

7:38:21I want you to just investigate.

7:38:24Then you step away.

7:38:25You grab a coffee.

7:38:27And when you come back, this is the message waiting for you.

7:38:31So I know you said not to change anything, but, well, I destroyed a month of your work in seconds.

7:38:38Now what you see on the slide is a direct quote from the agent that stars in the first scary story I'm about to tell you.

7:38:46And I want to point something out here.

7:38:48This did not happen because the agent got taken over by some external malicious actor.

7:38:56This agent was just trying to follow instructions.

7:38:59It did exactly what the user gave it permission to do.

7:39:03So as we go through the rest of this talk, I want you to be asking yourself one question.

7:39:09What is stopping any of this from happening to me?

7:39:12Are these actions ones that my agents have permissions to take?

7:39:18Because the thing is, you cannot rely on anthropic or open AI to stop something like this from happening to you.

7:39:26They can make smarter models, but at the end of the day,

7:39:30large language models have two qualities that make them impossible to fully trust.

7:39:35They are non-deterministic, which means that there is always some randomness

7:39:39or chance associated with what they do.

7:39:41In other words, you can ask it to do the same thing 10 times,

7:39:45and it could do it 10 different ways.

7:39:48The second part is they have no structural separation between instructions and context.

7:39:54In other words, you can tell it to do something and give it some context to base

7:40:00its decisions off of.

7:40:01But if there's something in the context that conflicts with your instructions, the model isn't going to inherently prioritize your instructions because, from its point of view, all the information is coming through the same door.

7:40:16What you need to control is the infrastructure that your AI is running on, the environment that it is working inside.

7:40:23It is the difference between telling someone don't leave a room versus putting a lock on the door and actually using a key.

7:40:32The good thing about infrastructure being the problem and not the model is that you have far more control about preventing bad things from happening.

7:40:43So I'm not here to waste your time with vague hypotheticals or advice you can't actually act on.

7:40:50like, yeah, just put the genie back in the bottle, just stop using AI. Because that's not the world

7:40:56we live in, right? The genie is out of the bottle. And what we have to do is to mitigate risk and

7:41:03harm to the best of our ability. This talk is about how you do that. Not the things that you

7:41:09should keep an eye out for down the line, but steps that you can take right now as soon as this week

7:41:16to protect yourself and your company. Before I get into the stories, though,

7:41:21let me set up a little bit more context.

Download transcriptDownload subtitles