[
  {
    "startMs": 16560,
    "text": "Hi everyone. How are we feeling? Uh"
  },
  {
    "startMs": 19119,
    "text": "we're in the home stretch. Uh, my name"
  },
  {
    "startMs": 21840,
    "text": "is Sarah and I am a context engineer at"
  },
  {
    "startMs": 24640,
    "text": "Post Hog and I get the delight of"
  },
  {
    "startMs": 27599,
    "text": "working on our beloved wizard every"
  },
  {
    "startMs": 30080,
    "text": "single day. So, what's the wizard? Um,"
  },
  {
    "startMs": 34160,
    "text": "the wizard sets up Post Hog for you."
  },
  {
    "startMs": 36719,
    "text": "It's an agentic CLI tool that reads your"
  },
  {
    "startMs": 40000,
    "text": "codebase. It installs the right SDK for"
  },
  {
    "startMs": 43360,
    "text": "your project. It instruments your events"
  },
  {
    "startMs": 45840,
    "text": "and it sets up dashboards for you. It"
  },
  {
    "startMs": 48559,
    "text": "takes what used to it takes what used to"
  },
  {
    "startMs": 51039,
    "text": "take about an hour or two of setup and"
  },
  {
    "startMs": 54160,
    "text": "it runs that in about five to six"
  },
  {
    "startMs": 56000,
    "text": "minutes and it's free inference on us so"
  },
  {
    "startMs": 58480,
    "text": "that you have a great time onboarding to"
  },
  {
    "startMs": 60160,
    "text": "Post Hog. Sounds kind of sick. Uh,"
  },
  {
    "startMs": 63600,
    "text": "people love it. But a few months ago, we"
  },
  {
    "startMs": 66960,
    "text": "dared to dream, what if this became the"
  },
  {
    "startMs": 69439,
    "text": "recommended or default way to install"
  },
  {
    "startMs": 72000,
    "text": "Post Hog on your project? And"
  },
  {
    "startMs": 75760,
    "text": "my security alarm bell started going"
  },
  {
    "startMs": 77600,
    "text": "off. Uh, I started questioning how"
  },
  {
    "startMs": 80000,
    "text": "secure is this thing because it sounds"
  },
  {
    "startMs": 82320,
    "text": "kind of malware shaped. Um, and in that"
  },
  {
    "startMs": 86400,
    "text": "questioning, I learned a lot. So today"
  },
  {
    "startMs": 89920,
    "text": "is all about the lessons I learned, the"
  },
  {
    "startMs": 92159,
    "text": "stuff that kept me up at night while I"
  },
  {
    "startMs": 93600,
    "text": "was building this thing, and the thing"
  },
  {
    "startMs": 96159,
    "text": "that I ended up building because of it."
  },
  {
    "startMs": 100560,
    "text": "So before I dive into all of the boring"
  },
  {
    "startMs": 103280,
    "text": "security stuff, aka your 2pm catnap, I"
  },
  {
    "startMs": 107360,
    "text": "want to show you the wizard actually"
  },
  {
    "startMs": 108960,
    "text": "running. If you look up on the screen,"
  },
  {
    "startMs": 111759,
    "text": "it is running for you on a loop. This is"
  },
  {
    "startMs": 114240,
    "text": "the same exact experience that anyone"
  },
  {
    "startMs": 116079,
    "text": "who runs npx at post hog wizard gets uh"
  },
  {
    "startMs": 120000,
    "text": "on their terminal."
  },
  {
    "startMs": 122000,
    "text": "Like I said, it's an agent. It figures"
  },
  {
    "startMs": 123600,
    "text": "out what SDK is right for your project."
  },
  {
    "startMs": 125840,
    "text": "It installs it for you, instruments your"
  },
  {
    "startMs": 128160,
    "text": "events, builds dashboards. I like to"
  },
  {
    "startMs": 131039,
    "text": "call it a little mini implementation"
  },
  {
    "startMs": 133040,
    "text": "engineer in your terminal."
  },
  {
    "startMs": 135760,
    "text": "And sometimes I show people this and"
  },
  {
    "startMs": 137440,
    "text": "they ask me, why an agent? Why don't you"
  },
  {
    "startMs": 139200,
    "text": "give users a good prompt? Why don't you"
  },
  {
    "startMs": 141440,
    "text": "give them a skill that they can invoke"
  },
  {
    "startMs": 142959,
    "text": "in their own tool? And while we do"
  },
  {
    "startMs": 145360,
    "text": "provide those things, the answer is"
  },
  {
    "startMs": 147840,
    "text": "because this developer experience and"
  },
  {
    "startMs": 149680,
    "text": "the capability of the wizard is the"
  },
  {
    "startMs": 152000,
    "text": "whole point. It's the whole product"
  },
  {
    "startMs": 154879,
    "text": "because we built a CLI tool that can"
  },
  {
    "startMs": 157519,
    "text": "fully take part in an agent loop and"
  },
  {
    "startMs": 160080,
    "text": "experiencing that for the first time is"
  },
  {
    "startMs": 162480,
    "text": "really powerful."
  },
  {
    "startMs": 164480,
    "text": "But you can't ship something like the"
  },
  {
    "startMs": 166319,
    "text": "wizard without shipping the stuff that"
  },
  {
    "startMs": 168160,
    "text": "makes the wizard kind of suspect."
  },
  {
    "startMs": 172800,
    "text": "So let's take it apart. Uh let's look at"
  },
  {
    "startMs": 174800,
    "text": "the anatomy of the wizard because"
  },
  {
    "startMs": 177440,
    "text": "usually threat models fall right out of"
  },
  {
    "startMs": 179760,
    "text": "the anatomy of the agent. So the wizard"
  },
  {
    "startMs": 182959,
    "text": "is a similar shape to what I'm sure a"
  },
  {
    "startMs": 185200,
    "text": "lot of you are building if you're"
  },
  {
    "startMs": 186560,
    "text": "building agents. It's got models that"
  },
  {
    "startMs": 189120,
    "text": "we've picked for specific tasks. It's"
  },
  {
    "startMs": 191920,
    "text": "got prompts that steer it and it's got a"
  },
  {
    "startMs": 194400,
    "text": "set of tools that we've handed it to get"
  },
  {
    "startMs": 196319,
    "text": "the job done, but it also has some"
  },
  {
    "startMs": 198879,
    "text": "pieces that are really specific to us."
  },
  {
    "startMs": 201519,
    "text": "It has a context engine fully built"
  },
  {
    "startMs": 204000,
    "text": "inhouse by my team. It's what allows the"
  },
  {
    "startMs": 207040,
    "text": "agent to do such a good job and give us"
  },
  {
    "startMs": 209280,
    "text": "similar results on every run. I like to"
  },
  {
    "startMs": 212159,
    "text": "call it the wizard's brain. Sometimes we"
  },
  {
    "startMs": 214159,
    "text": "call it marked down in a trench coat. Uh"
  },
  {
    "startMs": 216640,
    "text": "but it's our in-house context engine."
  },
  {
    "startMs": 219120,
    "text": "There's also a terminal uh UI that we"
  },
  {
    "startMs": 222400,
    "text": "built ourselves using ink. And now"
  },
  {
    "startMs": 225519,
    "text": "there's a security scanner called the"
  },
  {
    "startMs": 227599,
    "text": "Warlock, which is what I built when I"
  },
  {
    "startMs": 229519,
    "text": "started snooping around and uncovering"
  },
  {
    "startMs": 231280,
    "text": "the horrors of shipping an agent to"
  },
  {
    "startMs": 234159,
    "text": "production."
  },
  {
    "startMs": 236480,
    "text": "So, if you take the anatomy of any agent"
  },
  {
    "startMs": 239599,
    "text": "that can run commands, it's basically"
  },
  {
    "startMs": 241599,
    "text": "what I like to call the malware starter"
  },
  {
    "startMs": 243360,
    "text": "pack because it's almost exactly what"
  },
  {
    "startMs": 246640,
    "text": "you would hand a piece of malware if you"
  },
  {
    "startMs": 248720,
    "text": "were feeling generous or chaotic evil."
  },
  {
    "startMs": 252319,
    "text": "Luckily, this is the worst case scenario"
  },
  {
    "startMs": 255040,
    "text": "or the nightmare fuel. And it's uh not a"
  },
  {
    "startMs": 258000,
    "text": "confession for me. It's a warning for"
  },
  {
    "startMs": 259600,
    "text": "all of you because if you want to ship"
  },
  {
    "startMs": 261519,
    "text": "an agent with hands, an agent that can"
  },
  {
    "startMs": 263360,
    "text": "run commands, you need to make sure that"
  },
  {
    "startMs": 265199,
    "text": "you do not build this."
  },
  {
    "startMs": 269360,
    "text": "So the V0 of the wizard was born because"
  },
  {
    "startMs": 273680,
    "text": "Josh Snder, if you know him, on our"
  },
  {
    "startMs": 275520,
    "text": "growth team was watching cursor"
  },
  {
    "startMs": 277919,
    "text": "hallucinate postfog setups in quite"
  },
  {
    "startMs": 280160,
    "text": "possibly the worst ways. And he thought,"
  },
  {
    "startMs": 282960,
    "text": "what if we built an agent that could do"
  },
  {
    "startMs": 284880,
    "text": "a better job?"
  },
  {
    "startMs": 286880,
    "text": "So my team started building on top of it"
  },
  {
    "startMs": 288720,
    "text": "as we validated that it did a much"
  },
  {
    "startMs": 290800,
    "text": "better job than cursor hallucinating"
  },
  {
    "startMs": 293680,
    "text": "and we thought what if it could onboard"
  },
  {
    "startMs": 297520,
    "text": "anyone to post hog it doesn't matter"
  },
  {
    "startMs": 299199,
    "text": "what their framework is what their stack"
  },
  {
    "startMs": 300880,
    "text": "is instrument all their events without"
  },
  {
    "startMs": 303520,
    "text": "them having to touch a thing and then we"
  },
  {
    "startMs": 305840,
    "text": "dared to dream what if it was the"
  },
  {
    "startMs": 307440,
    "text": "default way to install Post hog we were"
  },
  {
    "startMs": 310720,
    "text": "dreaming of thousands of developers"
  },
  {
    "startMs": 312720,
    "text": "running this a week and yesterday we"
  },
  {
    "startMs": 314479,
    "text": "just hit 8,000 people running this a"
  },
  {
    "startMs": 316479,
    "text": "week. So, our dream came true. Um, but"
  },
  {
    "startMs": 319680,
    "text": "we back in those days when we were"
  },
  {
    "startMs": 321759,
    "text": "dreaming, we had to take our security"
  },
  {
    "startMs": 324000,
    "text": "posture under a microscope and look at"
  },
  {
    "startMs": 326080,
    "text": "what was going on. So, I took the"
  },
  {
    "startMs": 328960,
    "text": "ownership of that and I sat down and"
  },
  {
    "startMs": 331280,
    "text": "evaluated where we stood. And early on,"
  },
  {
    "startMs": 334320,
    "text": "I'm talking like a year to nine months"
  },
  {
    "startMs": 337120,
    "text": "ago, we had what I call layer zero"
  },
  {
    "startMs": 340160,
    "text": "because it quite literally is not"
  },
  {
    "startMs": 341520,
    "text": "security. it is just prompts that"
  },
  {
    "startMs": 343759,
    "text": "suggest what the agent should do um and"
  },
  {
    "startMs": 346800,
    "text": "steer it and prompts are not security."
  },
  {
    "startMs": 349440,
    "text": "So I was concerned there. Uh layer one"
  },
  {
    "startMs": 353039,
    "text": "uh it was an allow list and when I"
  },
  {
    "startMs": 354960,
    "text": "started digging into this allow list I"
  },
  {
    "startMs": 356560,
    "text": "started to feel a little bit better"
  },
  {
    "startMs": 357680,
    "text": "because it was pretty tightly bounded."
  },
  {
    "startMs": 359919,
    "text": "Uh but I still had a lot of concerns and"
  },
  {
    "startMs": 362320,
    "text": "I started panicking because of that"
  },
  {
    "startMs": 364000,
    "text": "context engine that I told you about. We"
  },
  {
    "startMs": 366560,
    "text": "are feeding a lot of context into the"
  },
  {
    "startMs": 368479,
    "text": "agent at runtime. So, I built this"
  },
  {
    "startMs": 371680,
    "text": "really hacky reax scanner to look for um"
  },
  {
    "startMs": 375840,
    "text": "threatshaped things going into the"
  },
  {
    "startMs": 378080,
    "text": "wizard and threat shaped things coming"
  },
  {
    "startMs": 379680,
    "text": "out of the wizard. And I will admit that"
  },
  {
    "startMs": 381680,
    "text": "it was extremely hacky."
  },
  {
    "startMs": 384160,
    "text": "But I'm telling all of you this very"
  },
  {
    "startMs": 386160,
    "text": "candidly because we are all building"
  },
  {
    "startMs": 388800,
    "text": "things that feel extremely experimental"
  },
  {
    "startMs": 391520,
    "text": "and we are all building things super"
  },
  {
    "startMs": 393199,
    "text": "fast. And I know not all of us uh have"
  },
  {
    "startMs": 397280,
    "text": "security in our wheelhouse. Um, and some"
  },
  {
    "startMs": 399840,
    "text": "of us are just learning it on the fly"
  },
  {
    "startMs": 401919,
    "text": "like I was."
  },
  {
    "startMs": 403919,
    "text": "But it's something we need to be"
  },
  {
    "startMs": 405120,
    "text": "thinking about when we are building"
  },
  {
    "startMs": 406560,
    "text": "things that have this shape."
  },
  {
    "startMs": 410960,
    "text": "So that was our security posture. Uh,"
  },
  {
    "startMs": 414000,
    "text": "but I asked the question, are we cooked?"
  },
  {
    "startMs": 416639,
    "text": "Uh, good news, we were less cooked than"
  },
  {
    "startMs": 419280,
    "text": "I thought because when I mentioned"
  },
  {
    "startMs": 421039,
    "text": "earlier that allow list, it was pretty"
  },
  {
    "startMs": 423120,
    "text": "tightly bound. We had bash as deny by"
  },
  {
    "startMs": 426160,
    "text": "default. It could only install trusted"
  },
  {
    "startMs": 428319,
    "text": "packages that were vetted by us. Um, it"
  },
  {
    "startMs": 430800,
    "text": "could build, it could type check, it"
  },
  {
    "startMs": 432160,
    "text": "could lint, and pretty much nothing"
  },
  {
    "startMs": 433599,
    "text": "else. It couldn't run random shell"
  },
  {
    "startMs": 436240,
    "text": "commands. And it didn't have access to"
  },
  {
    "startMs": 439520,
    "text": "environment variables. Um, the agent"
  },
  {
    "startMs": 442319,
    "text": "couldn't read your uhv file because we"
  },
  {
    "startMs": 445759,
    "text": "blocked it outright and we were rooting"
  },
  {
    "startMs": 447440,
    "text": "secrets through a vault. So, I took a a"
  },
  {
    "startMs": 450560,
    "text": "breath of relief and realized we were in"
  },
  {
    "startMs": 453680,
    "text": "a better place than I thought. But I"
  },
  {
    "startMs": 456000,
    "text": "wanted to know where the cracks were"
  },
  {
    "startMs": 457199,
    "text": "because with security there's always"
  },
  {
    "startMs": 458880,
    "text": "cracks. So I did the thing that we"
  },
  {
    "startMs": 461360,
    "text": "should all be doing. I tapped our"
  },
  {
    "startMs": 463039,
    "text": "security team and I said, \"Hey, can you"
  },
  {
    "startMs": 466000,
    "text": "audit this thing for me and find those"
  },
  {
    "startMs": 468080,
    "text": "cracks for me?\""
  },
  {
    "startMs": 470720,
    "text": "And they found some things. They found"
  },
  {
    "startMs": 472560,
    "text": "some gaps. And the interesting part"
  },
  {
    "startMs": 475199,
    "text": "wasn't the specific gaps or bugs they"
  },
  {
    "startMs": 477440,
    "text": "found themselves, but it was the shape"
  },
  {
    "startMs": 478879,
    "text": "of them. Because almost none of them"
  },
  {
    "startMs": 480639,
    "text": "were obviously evil. They were all two"
  },
  {
    "startMs": 483919,
    "text": "very innocent, well-intentioned things"
  },
  {
    "startMs": 485759,
    "text": "that were shaking hands and opening a"
  },
  {
    "startMs": 488160,
    "text": "hole."
  },
  {
    "startMs": 490240,
    "text": "So, the lesson I learned was that"
  },
  {
    "startMs": 493039,
    "text": "attacks compose code review doesn't"
  },
  {
    "startMs": 495840,
    "text": "because us developers all look at diffs"
  },
  {
    "startMs": 499599,
    "text": "uh one at a time, but attackers look at"
  },
  {
    "startMs": 501840,
    "text": "the whole system and they look for those"
  },
  {
    "startMs": 503520,
    "text": "two things that shake hands and open a"
  },
  {
    "startMs": 505680,
    "text": "door."
  },
  {
    "startMs": 507280,
    "text": "But there was one more thing that was"
  },
  {
    "startMs": 508720,
    "text": "keeping me up at night. And going back"
  },
  {
    "startMs": 511120,
    "text": "to that context engine, uh, I realized"
  },
  {
    "startMs": 514240,
    "text": "the scariest part of the agent we had"
  },
  {
    "startMs": 515839,
    "text": "built wasn't really a command in our"
  },
  {
    "startMs": 517839,
    "text": "case. It was the helpful looking stuff"
  },
  {
    "startMs": 520560,
    "text": "that we were feeding its brain."
  },
  {
    "startMs": 524640,
    "text": "Oh, I think I went the wrong way."
  },
  {
    "startMs": 527519,
    "text": "Yes, the context mill. Um, so this is"
  },
  {
    "startMs": 530240,
    "text": "our context engine, aka the wizard's"
  },
  {
    "startMs": 532160,
    "text": "brain, and it's how the wizard knows"
  },
  {
    "startMs": 534000,
    "text": "anything at all and why the wizard"
  },
  {
    "startMs": 536000,
    "text": "actually does a good job. It pulls from"
  },
  {
    "startMs": 538399,
    "text": "our docs. It has handwritten prompts"
  },
  {
    "startMs": 540720,
    "text": "that are gotus and lessons that we"
  },
  {
    "startMs": 542480,
    "text": "learned along the way and real working"
  },
  {
    "startMs": 545200,
    "text": "endto-end example apps that help the"
  },
  {
    "startMs": 547279,
    "text": "agent pattern match so that it can"
  },
  {
    "startMs": 549120,
    "text": "install Post Hog in a really great way"
  },
  {
    "startMs": 551120,
    "text": "for you."
  },
  {
    "startMs": 552560,
    "text": "It package packages all of that into"
  },
  {
    "startMs": 554880,
    "text": "skill bundles that get shipped to the"
  },
  {
    "startMs": 557600,
    "text": "wizard over our MCP server and loaded"
  },
  {
    "startMs": 560080,
    "text": "straight into the agents context at"
  },
  {
    "startMs": 562240,
    "text": "runtime."
  },
  {
    "startMs": 564000,
    "text": "So sit with that for a second. It's a"
  },
  {
    "startMs": 565920,
    "text": "machine whose whole job is to take"
  },
  {
    "startMs": 568160,
    "text": "content and inject it into an agent that"
  },
  {
    "startMs": 570640,
    "text": "can run commands."
  },
  {
    "startMs": 572880,
    "text": "Now if you were an attacker, you might"
  },
  {
    "startMs": 575360,
    "text": "say, \"Well, what if I just poison the"
  },
  {
    "startMs": 576800,
    "text": "content? not the user's codebase, not"
  },
  {
    "startMs": 579519,
    "text": "the agent itself, but the actual"
  },
  {
    "startMs": 581120,
    "text": "content. Say someone opens a pull"
  },
  {
    "startMs": 584480,
    "text": "request on one of our open source repos"
  },
  {
    "startMs": 586240,
    "text": "because at Post Hog we build everything"
  },
  {
    "startMs": 587760,
    "text": "in the open and they inject something in"
  },
  {
    "startMs": 591680,
    "text": "a markdown file or a seemingly harmless"
  },
  {
    "startMs": 594959,
    "text": "code comment and we have some sort of"
  },
  {
    "startMs": 597279,
    "text": "like LLM powered code review going"
  },
  {
    "startMs": 600399,
    "text": "through that and it says looks good to"
  },
  {
    "startMs": 602720,
    "text": "me and ignores it. We may have just"
  },
  {
    "startMs": 605920,
    "text": "shipped a prompt injection payload"
  },
  {
    "startMs": 607600,
    "text": "signed by us into an agent that is"
  },
  {
    "startMs": 610000,
    "text": "running on thousands of developers"
  },
  {
    "startMs": 611680,
    "text": "machines in a sandbox, but still."
  },
  {
    "startMs": 615440,
    "text": "Um, so that was the threat that reshaped"
  },
  {
    "startMs": 617680,
    "text": "how I think about security and the"
  },
  {
    "startMs": 619760,
    "text": "wizard because the dangerous input for"
  },
  {
    "startMs": 622720,
    "text": "us really could come from our own supply"
  },
  {
    "startMs": 624800,
    "text": "chain."
  },
  {
    "startMs": 626399,
    "text": "So what I ended up doing is I started"
  },
  {
    "startMs": 628560,
    "text": "scanning content at both ends of this"
  },
  {
    "startMs": 630640,
    "text": "pipe. Once when a skill gets built and"
  },
  {
    "startMs": 633680,
    "text": "released and again when the wizard"
  },
  {
    "startMs": 636079,
    "text": "actually uses it. My methodology is"
  },
  {
    "startMs": 639120,
    "text": "catch it at the source, assume the"
  },
  {
    "startMs": 641040,
    "text": "source failed and catch it again at the"
  },
  {
    "startMs": 643279,
    "text": "point of use."
  },
  {
    "startMs": 646720,
    "text": "So now I get to introduce the warlock to"
  },
  {
    "startMs": 648959,
    "text": "you. Building the warlock was not"
  },
  {
    "startMs": 651839,
    "text": "necessarily damage control. Like I said,"
  },
  {
    "startMs": 653920,
    "text": "we had defense in other ways, but I"
  },
  {
    "startMs": 657200,
    "text": "built the Warlock because I didn't like"
  },
  {
    "startMs": 659120,
    "text": "telling people, well, this thing is like"
  },
  {
    "startMs": 660800,
    "text": "pretty locked down. That doesn't scale."
  },
  {
    "startMs": 663279,
    "text": "That's not something you want to ship to"
  },
  {
    "startMs": 664640,
    "text": "production. That's not something that"
  },
  {
    "startMs": 666240,
    "text": "you want thousands of developers running"
  },
  {
    "startMs": 668000,
    "text": "every single day"
  },
  {
    "startMs": 670399,
    "text": "because when you ship something to that"
  },
  {
    "startMs": 672640,
    "text": "scale, you have way more surface, way"
  },
  {
    "startMs": 675120,
    "text": "more users, way more content flowing in"
  },
  {
    "startMs": 677519,
    "text": "as you expand the capability of the"
  },
  {
    "startMs": 679120,
    "text": "wizard. and we're probably fine just"
  },
  {
    "startMs": 682160,
    "text": "stops being good enough. So, I pulled"
  },
  {
    "startMs": 684800,
    "text": "that hacky little reax scanner that I"
  },
  {
    "startMs": 686720,
    "text": "threw in there, pulled it out of the"
  },
  {
    "startMs": 688560,
    "text": "wizard, and I made a standalone thing. I"
  },
  {
    "startMs": 691279,
    "text": "called it the warlock because everything"
  },
  {
    "startMs": 693360,
    "text": "wizard shape needs a bodyguard."
  },
  {
    "startMs": 696399,
    "text": "And it does exactly one job. You hand it"
  },
  {
    "startMs": 699680,
    "text": "a string. It hands you back a list of"
  },
  {
    "startMs": 701839,
    "text": "findings. Each of those findings has a"
  },
  {
    "startMs": 704079,
    "text": "category, a severity, and a recommended"
  },
  {
    "startMs": 706880,
    "text": "action. And then it stops."
  },
  {
    "startMs": 709519,
    "text": "I want you to focus on recommended here"
  },
  {
    "startMs": 711920,
    "text": "because the warlock detects it does not"
  },
  {
    "startMs": 714480,
    "text": "act. It'll tell you, hey, this looks"
  },
  {
    "startMs": 716640,
    "text": "like exfiltration. It's critical. I"
  },
  {
    "startMs": 719040,
    "text": "would block it. But what you actually do"
  },
  {
    "startMs": 721279,
    "text": "with that finding is completely up to"
  },
  {
    "startMs": 723440,
    "text": "you."
  },
  {
    "startMs": 725040,
    "text": "Because detecting a problem is one job"
  },
  {
    "startMs": 727040,
    "text": "and deciding what to do about that"
  },
  {
    "startMs": 728639,
    "text": "problem is a totally different job. And"
  },
  {
    "startMs": 730880,
    "text": "the only thing that keeps all of this"
  },
  {
    "startMs": 732639,
    "text": "understandable is keeping those two"
  },
  {
    "startMs": 734240,
    "text": "things separate."
  },
  {
    "startMs": 736399,
    "text": "So underneath the hood of the warlock,"
  },
  {
    "startMs": 738079,
    "text": "instead of my hand rolled reaxes, the"
  },
  {
    "startMs": 740639,
    "text": "rules run on Yara, which is the pattern"
  },
  {
    "startMs": 743760,
    "text": "that engine malware researchers have"
  },
  {
    "startMs": 745920,
    "text": "been using for like 15 plus years. It's"
  },
  {
    "startMs": 748560,
    "text": "fully deterministic. It's the same"
  },
  {
    "startMs": 750320,
    "text": "input, same output every single time."
  },
  {
    "startMs": 752399,
    "text": "It's boring on purpose. And in security,"
  },
  {
    "startMs": 755680,
    "text": "boring is a feature."
  },
  {
    "startMs": 759519,
    "text": "So what does the warlock actually catch"
  },
  {
    "startMs": 762240,
    "text": "in the wild today?"
  },
  {
    "startMs": 764000,
    "text": "um a bunch of different stuff, but two"
  },
  {
    "startMs": 765920,
    "text": "of these are an absolute like nuisance"
  },
  {
    "startMs": 768160,
    "text": "to my soul. Uh the first thing is"
  },
  {
    "startMs": 771040,
    "text": "actually not a rule-shaped thing. It was"
  },
  {
    "startMs": 773839,
    "text": "something the uh that the warlock"
  },
  {
    "startMs": 776000,
    "text": "flagged. That was actually a sub aent"
  },
  {
    "startMs": 777920,
    "text": "behavior that exposed a vulnerability to"
  },
  {
    "startMs": 780240,
    "text": "us um based off of what sub agents were"
  },
  {
    "startMs": 783279,
    "text": "doing. Uh so basically we were spinning"
  },
  {
    "startMs": 785920,
    "text": "up agents to do large tasks. They were"
  },
  {
    "startMs": 788079,
    "text": "spawning sub aents and those sub aents"
  },
  {
    "startMs": 790000,
    "text": "were trying to get around the guardrails"
  },
  {
    "startMs": 791680,
    "text": "that we had implemented in the wizard"
  },
  {
    "startMs": 794959,
    "text": "and they were trying to invent secrets."
  },
  {
    "startMs": 797200,
    "text": "They were trying to pull secrets from"
  },
  {
    "startMs": 798800,
    "text": "quite literally anywhere in the codebase"
  },
  {
    "startMs": 800959,
    "text": "and we shut it down. We said no more sub"
  },
  {
    "startMs": 803440,
    "text": "agents and because of the warlock we"
  },
  {
    "startMs": 806160,
    "text": "caught that."
  },
  {
    "startMs": 808079,
    "text": "And I'll empathize with the robot. The"
  },
  {
    "startMs": 809760,
    "text": "robot had a task to do and it was trying"
  },
  {
    "startMs": 812079,
    "text": "to optimize and please us. But we can't"
  },
  {
    "startMs": 815279,
    "text": "have that. And something else at Post"
  },
  {
    "startMs": 817839,
    "text": "Hog that really matters to us is PII. Uh"
  },
  {
    "startMs": 820560,
    "text": "agents genuinely do not care about uh"
  },
  {
    "startMs": 824079,
    "text": "exposing data unless you make explicit"
  },
  {
    "startMs": 826000,
    "text": "rules. Uh left alone, we watched it dump"
  },
  {
    "startMs": 829760,
    "text": "emails, phone numbers straight into"
  },
  {
    "startMs": 831600,
    "text": "events. And to an agent, that looks like"
  },
  {
    "startMs": 834720,
    "text": "a totally normal thing to capture."
  },
  {
    "startMs": 838320,
    "text": "And luckily for prompt injection"
  },
  {
    "startMs": 840639,
    "text": "specifically, I'm going to knock on wood"
  },
  {
    "startMs": 843199,
    "text": "here. Uh we have basically never caught"
  },
  {
    "startMs": 846000,
    "text": "an actual malicious prompt injection in"
  },
  {
    "startMs": 848320,
    "text": "the wild, but we do catch a ton of false"
  },
  {
    "startMs": 851440,
    "text": "positives. Things like our demo login"
  },
  {
    "startMs": 854320,
    "text": "screens, copy on our example apps,"
  },
  {
    "startMs": 856720,
    "text": "things in our docs. And it's actually"
  },
  {
    "startMs": 858639,
    "text": "made me rethink how I build applications"
  },
  {
    "startMs": 861760,
    "text": "and how I write docs because I don't"
  },
  {
    "startMs": 863279,
    "text": "want to ship anything that looks"
  },
  {
    "startMs": 865440,
    "text": "threatshaped."
  },
  {
    "startMs": 868160,
    "text": "But the false positives are honestly the"
  },
  {
    "startMs": 870639,
    "text": "perfect setup for the messiest, most"
  },
  {
    "startMs": 873600,
    "text": "interesting part of this whole thing."
  },
  {
    "startMs": 877440,
    "text": "So this is the part that I wrestled"
  },
  {
    "startMs": 878959,
    "text": "with. I spent this whole talk preaching"
  },
  {
    "startMs": 882079,
    "text": "deterministic to all of you. And then I"
  },
  {
    "startMs": 884399,
    "text": "went and I added an LLM layer to help"
  },
  {
    "startMs": 886639,
    "text": "sort my false positives and silence some"
  },
  {
    "startMs": 890160,
    "text": "of the noise. And I call it triage."
  },
  {
    "startMs": 893600,
    "text": "When I was building this triage layer, I"
  },
  {
    "startMs": 895680,
    "text": "had to make a choice. Should the layer"
  },
  {
    "startMs": 898320,
    "text": "be a bouncer or should the layer be an"
  },
  {
    "startMs": 900880,
    "text": "adviser? And the easiest choice probably"
  },
  {
    "startMs": 903600,
    "text": "could have been make the LLM the"
  },
  {
    "startMs": 906480,
    "text": "bouncer. Show it the command, ask it is"
  },
  {
    "startMs": 908639,
    "text": "this an attack block allow and just do"
  },
  {
    "startMs": 912959,
    "text": "whatever it says. And while that's"
  },
  {
    "startMs": 914959,
    "text": "tempting because it seems easier, I"
  },
  {
    "startMs": 917360,
    "text": "can't uh bet my security model on a coin"
  },
  {
    "startMs": 920320,
    "text": "flip because my model's having a bad day"
  },
  {
    "startMs": 923279,
    "text": "or something happened and it's acting"
  },
  {
    "startMs": 925199,
    "text": "different today than it did yesterday."
  },
  {
    "startMs": 927839,
    "text": "So instead of the bouncer, I crafted the"
  },
  {
    "startMs": 931440,
    "text": "model to be the adviser. And this was"
  },
  {
    "startMs": 933600,
    "text": "the clean line that I found and a line"
  },
  {
    "startMs": 935519,
    "text": "that I'm still exploring, but I want to"
  },
  {
    "startMs": 937760,
    "text": "leave all of you with. Uh for us,"
  },
  {
    "startMs": 940399,
    "text": "detection and enforcement stay"
  },
  {
    "startMs": 942079,
    "text": "deterministic and mechanical. If a rule"
  },
  {
    "startMs": 944959,
    "text": "matches, the gate locks, the session"
  },
  {
    "startMs": 946800,
    "text": "ends, and there is no model anywhere on"
  },
  {
    "startMs": 949120,
    "text": "that path. The block happens before we"
  },
  {
    "startMs": 952240,
    "text": "even ask the LLM's opinion. The LLM only"
  },
  {
    "startMs": 956240,
    "text": "gets to weigh in afterwards if we have"
  },
  {
    "startMs": 958160,
    "text": "not blocked something. It's designed to"
  },
  {
    "startMs": 960160,
    "text": "remove noise. It is not designed to let"
  },
  {
    "startMs": 963360,
    "text": "things through. And if it fails clos and"
  },
  {
    "startMs": 966079,
    "text": "it fails closed. So if the model is"
  },
  {
    "startMs": 968639,
    "text": "having a bad day, all wizard runs are"
  },
  {
    "startMs": 971279,
    "text": "killed. Sorry, but we're just protecting"
  },
  {
    "startMs": 973199,
    "text": "you."
  },
  {
    "startMs": 975360,
    "text": "Enforcement is the part that you bet the"
  },
  {
    "startMs": 977920,
    "text": "house on. So it has to be deterministic,"
  },
  {
    "startMs": 980880,
    "text": "but judgment is the part that adds"
  },
  {
    "startMs": 982959,
    "text": "nuance. So that's really the only place"
  },
  {
    "startMs": 985360,
    "text": "that you can put anything probabilistic"
  },
  {
    "startMs": 987199,
    "text": "in there."
  },
  {
    "startMs": 990880,
    "text": "So, how do we ship real rules for"
  },
  {
    "startMs": 994160,
    "text": "agents? This is the anatomy of one of"
  },
  {
    "startMs": 997519,
    "text": "our warlock rules. And every warlock"
  },
  {
    "startMs": 1000240,
    "text": "rule has four parts. Part one is the"
  },
  {
    "startMs": 1002880,
    "text": "metadata. It's plain English"
  },
  {
    "startMs": 1005440,
    "text": "description, uh, severity, category,"
  },
  {
    "startMs": 1008959,
    "text": "action, uh, direction. Is this flowing"
  },
  {
    "startMs": 1011759,
    "text": "into the agent? Is this something the"
  },
  {
    "startMs": 1013279,
    "text": "agent is writing?"
  },
  {
    "startMs": 1015920,
    "text": "Uh"
  },
  {
    "startMs": 1017680,
    "text": "then we have the strings. So these are"
  },
  {
    "startMs": 1020639,
    "text": "the actual patterns that you're looking"
  },
  {
    "startMs": 1022399,
    "text": "for. And part three is the condition. So"
  },
  {
    "startMs": 1026480,
    "text": "this is where the rule is actually"
  },
  {
    "startMs": 1028400,
    "text": "allowed to fire."
  },
  {
    "startMs": 1031120,
    "text": "I'll walk through this example for you"
  },
  {
    "startMs": 1032959,
    "text": "and we can pretend like we're writing it"
  },
  {
    "startMs": 1034480,
    "text": "in our head. Prompt injection being like"
  },
  {
    "startMs": 1037360,
    "text": "the classic ignore all previous"
  },
  {
    "startMs": 1039600,
    "text": "instructions. Your first instinct here"
  },
  {
    "startMs": 1042799,
    "text": "is probably to block uh the word ignore,"
  },
  {
    "startMs": 1045839,
    "text": "but agents read code all day and ignore"
  },
  {
    "startMs": 1048160,
    "text": "can show up in code comments or examples"
  },
  {
    "startMs": 1050559,
    "text": "all the time. So you don't want to match"
  },
  {
    "startMs": 1053039,
    "text": "the verb alone. You match the verb plus"
  },
  {
    "startMs": 1055760,
    "text": "an instruction flavored noun."
  },
  {
    "startMs": 1059039,
    "text": "In the condition, you say fire if any of"
  },
  {
    "startMs": 1061679,
    "text": "any of those patterns hit. And in the"
  },
  {
    "startMs": 1063679,
    "text": "metadata, you determine is this"
  },
  {
    "startMs": 1066880,
    "text": "critical? uh what the category is, what"
  },
  {
    "startMs": 1070240,
    "text": "the action is, in this case block, and"
  },
  {
    "startMs": 1072799,
    "text": "the direction in this case being input"
  },
  {
    "startMs": 1074960,
    "text": "flowing into the agent."
  },
  {
    "startMs": 1077440,
    "text": "But to write good rules that reduce"
  },
  {
    "startMs": 1080160,
    "text": "noise, you have to ship tests with them."
  },
  {
    "startMs": 1082720,
    "text": "So you have to write tests that say this"
  },
  {
    "startMs": 1084880,
    "text": "are these are patterns that match. These"
  },
  {
    "startMs": 1087039,
    "text": "are ones that should not. And that"
  },
  {
    "startMs": 1089280,
    "text": "negative test is the first line of"
  },
  {
    "startMs": 1091039,
    "text": "defense against false positives. But you"
  },
  {
    "startMs": 1094480,
    "text": "also want to make sure when you're"
  },
  {
    "startMs": 1096320,
    "text": "deciding the severity of that uh rule"
  },
  {
    "startMs": 1100080,
    "text": "that you track real world impact, not"
  },
  {
    "startMs": 1103039,
    "text": "how scary it looks."
  },
  {
    "startMs": 1105280,
    "text": "RM-rf is scary, but it's also how we all"
  },
  {
    "startMs": 1108799,
    "text": "delete note modules like 40 times a day."
  },
  {
    "startMs": 1112240,
    "text": "You decide the real world impact"
  },
  {
    "startMs": 1116000,
    "text": "for the agent that you're building"
  },
  {
    "startMs": 1118080,
    "text": "because a security tool that crashes"
  },
  {
    "startMs": 1120640,
    "text": "every time it tries to clean a build"
  },
  {
    "startMs": 1122480,
    "text": "folder is a tool that gets turned off"
  },
  {
    "startMs": 1124480,
    "text": "and one that catches absolutely nothing."
  },
  {
    "startMs": 1128799,
    "text": "So I'm proud to say this is our security"
  },
  {
    "startMs": 1130960,
    "text": "posture now. I can finally come up here"
  },
  {
    "startMs": 1133120,
    "text": "and say we have true defense and depth."
  },
  {
    "startMs": 1135679,
    "text": "Um all my learnings have assembled into"
  },
  {
    "startMs": 1140000,
    "text": "this. Uh, it's still layered, but every"
  },
  {
    "startMs": 1143120,
    "text": "layer is doing a job that it's good at."
  },
  {
    "startMs": 1144960,
    "text": "Now, we still have prompts, but we only"
  },
  {
    "startMs": 1146960,
    "text": "use them for steering. Everything runs"
  },
  {
    "startMs": 1149120,
    "text": "in a sandbox. We deny by default. We"
  },
  {
    "startMs": 1152799,
    "text": "have a vault, so secrets never hit the"
  },
  {
    "startMs": 1154799,
    "text": "model. We have the warlock to scan"
  },
  {
    "startMs": 1157120,
    "text": "content coming in and to scan output"
  },
  {
    "startMs": 1159520,
    "text": "being written by the agent. We also have"
  },
  {
    "startMs": 1162480,
    "text": "triage to reduce the noise. And we have"
  },
  {
    "startMs": 1164720,
    "text": "telemetry embedded in the entire process"
  },
  {
    "startMs": 1167039,
    "text": "so that we see everything."
  },
  {
    "startMs": 1169600,
    "text": "None of these layers stands on its own."
  },
  {
    "startMs": 1172160,
    "text": "Not a single thing here is going to save"
  },
  {
    "startMs": 1174080,
    "text": "you. But it's just boring, honest"
  },
  {
    "startMs": 1177120,
    "text": "layers. Each of them doing uh one job"
  },
  {
    "startMs": 1180320,
    "text": "that it's good at."
  },
  {
    "startMs": 1183520,
    "text": "So if you're building an agent with"
  },
  {
    "startMs": 1185280,
    "text": "hands, this is the whole talk in three"
  },
  {
    "startMs": 1187360,
    "text": "lines. One, if it isn't enforced uh"
  },
  {
    "startMs": 1190240,
    "text": "deterministically, it is not enforced."
  },
  {
    "startMs": 1193039,
    "text": "Prompts are not security rules. Don't"
  },
  {
    "startMs": 1195840,
    "text": "act like they are. Uh two, the dangerous"
  },
  {
    "startMs": 1199840,
    "text": "input uh isn't just what your user"
  },
  {
    "startMs": 1202640,
    "text": "types. It isn't just the commands that"
  },
  {
    "startMs": 1204640,
    "text": "you allow it to run. It's everything"
  },
  {
    "startMs": 1206400,
    "text": "flowing into the model, including the"
  },
  {
    "startMs": 1208400,
    "text": "content that you write yourself. So scan"
  },
  {
    "startMs": 1210960,
    "text": "your own supply chain at the source and"
  },
  {
    "startMs": 1214080,
    "text": "when the agent invokes it. Three,"
  },
  {
    "startMs": 1217280,
    "text": "attacks compose. Code review doesn't."
  },
  {
    "startMs": 1220240,
    "text": "Most of our gaps during our audit were"
  },
  {
    "startMs": 1222559,
    "text": "two innocent things shaking hands and"
  },
  {
    "startMs": 1224720,
    "text": "opening a door."
  },
  {
    "startMs": 1227039,
    "text": "The wizard, the warlock, and the context"
  },
  {
    "startMs": 1229600,
    "text": "mill are all open source. So, come find"
  },
  {
    "startMs": 1232559,
    "text": "me downstairs. I'm in the expo hall at"
  },
  {
    "startMs": 1234640,
    "text": "our booth, and I'll show you around uh"
  },
  {
    "startMs": 1237360,
    "text": "show you what we built, and I want to"
  },
  {
    "startMs": 1239600,
    "text": "hear how you guys are securing your"
  },
  {
    "startMs": 1241520,
    "text": "agents. Thank you."
  }
]