Exploits Now Beat Patches: Open Source Security in the AI Era - Patrick Smyth, Chainguard

So this is, uh, how Frontier AI changed risk tolerance for open source. Um, kinda wanted to subtitle it, you know, How I Learned to Love the Atomic Bomb or whatever, but, uh, decided not to go for that. So I'm Patrick Smyth. Um, I work for Chainguard. Um, when I'm not thinking about the software supply chain or messing around with, um, you know, frontier AI, uh, I am hanging out in Queens. I'm one of these annoying people who like saunas all the time. Uh, if you're out of town, we have some great ones in the city. There's Tenth Street, that's

one that's around since the nineteenth century, and they have great food. And then there's one in Fidi too. So, um, you know. And then I know there's a lot of finance per-- folks interested in this, so, uh, that's the one. Eighty-eight Fulton, that's the one if you're, if you're down there. Um, okay. So, uh, this is gonna be the scaring you part of the, the talk basically. But, um, but this year-- So here's some, just some fun headlines. Um, so a seventeen-year-old, uh, Osakan,

uh, teenager, he's still in high school, uh, used ChatGPT, wrote some scripts, um, pu-- uh, exfiltrated, um, something like four point-- four and a quarter million records from the biggest, um, internet cafe in Japan, and, uh, then used one of the cards to, to buy some Pokemon cards or-- u-used one of the credit cards to buy Pokemon cards or whatever. Um, so, you know, people can be very smart and very dumb at that age. Um, FunkSec. Um, somebody hits, uh, some-- thought to be

an Algerian semi-technical person, hit eighty-five, um, targets over the course of the last, uh, few months. Um, one individual who has not been identified, um, used Claude Code to attack, um, seventeen organizations from he-- ranging from healthcare to, uh, government, um, to, you know, major corporations. Used Claude for everything from writing the ransom letters, picking the next target, uh, organizing his files,

um, and of course, doing the attacks. Um, kind of a takeaway here is that, uh, way I've been thinking about it is everyone kinda goes a level up in terms of threat. So if you're an actual engineer before, now you're like a, a team. If you were actually, you know, a, a team before, now you're like advanced persistent threat level. Um, so here's a little, uh, another little, uh, tidbit for you. Um, this is a, uh, this is basically, uh, whether the patch,

uh, comes before the exploitation. So the exploit, you know, obviously is the attack, the patch is, uh, remediation of the vulnerability. Um, back in ancient history in twenty eighteen, twenty nineteen, it was sixty-three days. This is an average, so it's pretty noisy, um, but it tells you something. Um, it's kind of-- There was this like secular trend that was going on for a long time that I guess is continuing, though it's kinda been drowned out of like, uh, you could kind of think of it as, you know, digitalification or something like that. But basically, like every--

you know, hospitals were using Python code all of a sudden and stuff like that. So that secular trend sort of was responsible for this tightening over the course of, you know, twenty-one, twenty-two, kind of probably twenty-three. But then you see in twenty-three and twenty-four and last year, it just kind of left, right? I wonder what could've been causing all of that. Uh, we'll talk about it. Uh, I think you already know. Um, but now it's negative seven d-days. So actually, the exploit, the on average hits before the patch.

This is from Mandiant. So, uh, they have their M-Trends report. Um, this is kind of what it used to look like if you were at an organization who, uh, you know, you would ha-- Say, uh, you, you have X amount of software in your stack, you run scanners against it, um, such as, you know, scanners like Grype, Trivy, et cetera. Trivy also had a bad year. Um, you, uh, scan against it, you get your X number of CVEs, known vulnerabilities that are rated, you know, from critical, high, medium, low, um,

that goes to a team. They get some amount of time to remediate. All very sensible, all very good. Um, the only problem is thirty days may as well be a million years in terms of some of these attacks that have been happening. So this timeline, basically, you can't re-- you know. I mean, obviously you have to have some kind of timeline just because you're remediating on some cadence, but it basically doesn't work that well anymore. Um, between, uh, y-- attackers r-jumping on known vulnerabilities almost immediately, exploiting, finding

vulnerabilities out, out of band. Um, and now these new sources of zero days, cough, cough, we also know what that is. Um, you know, Mythos, new frontier models, um, that this timeline is basically completely out the window. Um, this is a whole other area of, uh, of fun and enjoyment that we had this, this last year. So this is, um, just these m-- I don't know what you'd call it, but like these are-- I kind of-- You could think of it as the four horsemen here. There's ma-- waves and waves of malware, um,

every month. Uh, you know, I used to do a meme calendar for, um, supply chain attacks in old presentations where I'd have like one a mo-- one big one every month. And then I used to be like, "A big attack every month, guys. It's, it's really bad out there." And now it's like, it would be crazy. I could do a meme calendar, uh, of every day pretty much. And but the-- You know, this is basically just-- I put, um, four up here, but three from one month. Um, and then Shai-Hulud, which of course is just a perennial source of

fun. But basically, what is Shai-Hulud? It's a huge worm, originated in the JavaScript ecosystem. Kind of showed everyone that you could do a huge worm and do really well. So now there's huge numbers of copycat Shai-Huluds. There's SANDWORM_MODE, all this kinda stuff. Shai-Hulud is a worm from Dune. Um, so you know, it's a worm, worm, very clever. And then, uh, some of the code of these things, if you look at, is actually pretty... I mean, you-- there shouldn't be too entertained by attackers, like comments in their code or whatever, but it-- they, they, they name things in amusing ways. Um, Trivy got attacked. That's a scanner. Um, this

is all-- The other three were all March this year. Um, Trivy, uh, LiteLLM, a, uh, kind of router for, for agents and, um, you know, Axios was only... A supply chain attack, it was only live for two hours, six hundred thousand, um, downloads, okay? And a lot of these supply chain attacks, so this is malware we're talking about in sp-in specific. A lot of these attacks are chaining one from the other because everyone has everyone else's dependencies. You att- compromise one person, now you're compromising other people.

So something's wrong in the state of Denmark. Um, this is from Hamlet 2000. It's a weird movie. You can go back and watch Hamlet 2000. Um, and, uh, yeah, it's agents, right? It's frontier agents. And I think everyone in the room kinda knew that, um, that they have been accelerating everything, but, but you, you put that salt and pepper on everything. You know, you're accelerating how you make code, you're accelerating everything. Um, but, but it also attackers are making heavy use of these. Um, and the environments have

changed a lot. Um, I don't know if folks have heard, you know, maybe I'm old or whatever, or at least I kind of am an enthusiast of early computing and stuff. There's this concept called the Eternal September. Basically, it was back in the day, there would be these, uh, university networks. People were very like, you know, people were probably wearing their bow ties or whatever. People were s- were very, um, polite. They had their, like, n- community norms. Um, everything would be stable. And then what would happen? September would roll around, students would join. Those accursed, uh, things that are

always clogging up the university, students would arrive and start messing up all the, the networks and everything like that. They wouldn't know any of the norms. They'd be talking in capital letters. They wouldn't know any of the good emoticons or whatever. Um, so another way of thinking about it is the Eternal September has kind of gone on forever, and this is ha- this happens all the time. Cultur-- It's a cultural thing, but it's also like, naturally you have an environment or community that is like high trust and moves to a, a sort of more adverse low trust situation, okay? Um,

another way of thinking about this is... This-- and this is just the basic problem of software supply chain security, okay? And it's very simple, but when you think this way, I think it kinda clarifies a lot, is that you have users. They're relying on your infrastructure, on your software, um, to do the things that they do with, with your s- with your, your stack, with your business. Um, you have a responsibility to them. Then you turn around, and you're actually pulling from upstream. You are pulling from, uh,

places that are, you know, you're trying to vet, you're trying to keep up with it, but it's difficult. It's a res-- And any of those things that you pull in could potentially compromise you and your users, okay? So it's kind of... But we're all in this position, right? We're all between-- in this very specific rock and a hard place. And this is the ba-- this is the fundamental problem of software supply chain security. Um, so again, we're gonna move from just random g- doom and gloom to actually what you have to and should do this year. But I'm just gonna throw out a few

things, but this is to make a larger point, okay? You have to do, um, if you're thinking about, uh, packages, you have to do cooldown periods. If you are, um, you ha-- If you're thinking about C- uh, CVEs, you have to remediate them faster than ever. Um, you have to sign, right? Um, I'll talk about it later, but SLSA and Sigstore, you know, sign your, your, uh, your artifacts. Um, you have to worry about operational security on developer laptops, um, including physical

keys. Um, there's a lot you have to worry about. Uh, the point here isn't so much... I think I'll dial in on a whole bunch of these as we go on. But the point here is actually, like, it's time to get serious. Like, it's actually time to, to take this seriously because right now it's not twenty twenty-four, it's not twenty twenty-three. It's a actually significantly and much more adverse environment out there than it was just a few years ago, okay? Um, there's a-- And there's a lot to do, and it costs, and, uh, it's painful. But, um,

so some of these you should do first. And also, I would say if you're still, you know, moving to containerize, if you're still moving to do CI/CD, um, great time to finish those projects, right? Um, becau-- or I would say you, you almost kinda need to, because if you don't have some of those, um, basic fundamental, uh, pieces in place, it can become very hard to do these other modernization efforts that have become more important over the last couple of years. Now, luckily, now you have AI to do some leapfrogging there, so I wouldn't be, you know, so

down. But time to get that skunk works going, time to get those projects going, time to complete, um, your golden images program, et cetera, et cetera, okay? Centralize your containers. Um, who are we? We're Chainguard. Um, we've been thinking about this since twenty twenty-one. Um, we have, uh, four founders. They've created, uh, projects. You have-- might have heard of them, Kubernetes, uh, Sigstore, SLSA, um, uh, Google Distroless. And we, you know, this is basically-- We're basically here to just, just think about

the software sup-supply chain security problem and try to, like, nuke it from orbit, basically. Um, you know, we're... I'll get into some of the specifics that we do, um, in a second, but here's so- just a few surfaces that I kinda wanna dial into, okay? So first of all, you've got, um... And these all interact in different ways, but you can think of them somewhat separately, and sometimes it's helpful to think of them separately, okay? On the one hand, you've got, um, malware, okay? So that was

like your Trivy, your Axios, everything like that. That's one vector, okay? That's the kind of thing a cooldown period is gonna help you with. Um, then you've got known vulnerabilities, okay? So this is kind of like legacy security. Um, yet people are often like, "Oh, C..." You know, it's-- CVE is kind of the unit of this, right? Common Vulnerability and Exposure. People are like, "CVEs are, are bullshit. CVEs are..." Yeah, a lot of CVEs are basically noise, but X percentage are actually important, you know, that you actually need to remediate. The rest make--

The rest noise makes it harder to deal with the ones you actually need to deal with. And because they're known vulnerabilities, you have to fix them because they're in a database, right? Like, someone could just be like Go find... Here, look at your software, look at the database, and combine those two things together, right? So it's actually very important to remediate known vulnerabilities. We now, even more importantly, have this realm of unknown vulnerabilities or like, uh, zero-days, okay? So this is becoming more important with the rise of frontier models that have quote, unquote, "the juice,"

like Mythos, um, ChatGPT now has Daybreak, um, but A-Astra also very capable in this, in this way. Um, basically, frontier models are very good at chaining together lots of smaller vulnerabilities to make exploits, okay? And, and so that realm of zero-days is very important. Um, then we have these emerging layers, or you could also think about them as like side targets, soft targets, side doors that attackers are entering in through, okay? Used to be they'd come to the front door and be like, "No admittance. Our production code is very secure."

Now attackers kinda know, "Hey, actually, let's hit your CI/CD 'cause that's-- no one's really looking at it that much. It's misconfigured. It's running a lot of dependencies and stuff in there. Uh, let's hit the developer workstation 'cause the developer is pulling things to prepare." You have a cooldown period, but does your developer workstation's laptop have the cooldown period? No. So let's attack the developer laptop. Let's attack the new agentic layer. It's emerging, it's changing a lot, people don't even know what's going on, and you can poison the models. So the, the-- that's these new, uh, side doors that attackers are very

conscious are the soft targets. And, and the last year has shown that. macOS stealers, all this kind of stuff. So this is kinda how Chainguard's thinking about it, and then we'll dial in on-- I'll try to also combine with more general ways that these things can be addressed, okay? For zero-days, we have, um, we've, uh, I'll actually get to Athena last. Um, so our classic, um, uh, a classic sort of thing that we've been providing for a long time is, um, containers that have no known vulnerabilities. So if you migrate onto them, they basically don't have any vulnerabilities at all. I'll talk more

about that in a second. Um, for malware, we have our Chainguard libraries. We build from source. Um, you pull from us instead of from PyPI, npm, Maven Central, and it makes a lot of attacks structurally impossible. Um, so basically replacement repositories. Um, for zero-days, we've been working with, um, major banks, um, and other, uh, tech companies and, uh, we have a large consortium going. T-Athena is a consortial solution, so basically we, we do have a clearing house

where we, um, take, uh, uh, known zero-days and then we provide them back to people within the consortium. Um, and then we also have our new, uh, a- Chainguard Actions and Chainguard ski-- uh, Chainguard Agent Skills that are hardened, uh, versions of CI/CD actions or agent skills, um, that you can kinda drop in. Um, in terms of the, the sort of that area that I was talking about, that CVEs area, that sort of like legacy area that you still can't ignore, uh,

the big problem is this like iceberg problem, right? Is that you have to run on some infra, but that infra is, um, most of it is beyond your control. Um, so that basically means that like, say, two percent of the code that you run is your application code. That's a made-up number, but I would say it's actually not a bad made-up number. It's actually probably pretty close. Could be one percent, um, is actually your code that your engineers wrote, and they can introduce problems in that. That's not-- We do software supply chain security. Everything else is kind of more our area. So

if you con-- don't control the OS, um, the application layer, the language runtime, that-- those are the layers that most of the vulnerabilities are on, okay? And I call this the, uh, my boss told me to fix Ubuntu problem. Um, and it basically is like, well, okay, now we somehow have to fix all these things in systems that we don't control. The way that Chainguard thinks about this is that we actually started over with a new distribution of Linux that is called Wolfi, and it's basically just completely, um, you know, from the-- built from the ground up to be--

to solve this problem. And we may mostly do that by having it be a rolling distro. So there's kind of like not much of an upstream beyond where you're basically running on upstream maintainers', um, code, okay? And we pull in very quickly. Things get pulled in. Images get re-rebuilt, uh, one to four times a day. Um, so it's a very different way of approach. We don't sit on stable versions, okay? It's a very different approach. We also try to be minimal and do some other things. Minimal also helps with zero-days. This is our SLA. So no thirty/sixty/ninety. It's, uh,

one/seven/fourteen. One day for a known exploited vulnerability, seven days for criticals, and then fourteen days for everything else. So the fourteen, for a lot of that fourteen, that's stuff that no team would ever fix, right? Like, a lot, on a lot of teams, mediums and lows would never be triaged in under any circumstances. So, um, what else do, do we think is really important? This is not a product. This is, uh, something we care about. Um, SLSA and Sigstore. Um, basically, we think you should sign everything.

Whenever an artifact is created, sign it and then pass it off, then check that signature on the other end, okay? That's-- And what you can use for this, for this stack is Sigstore is very useful for this, okay? Um, a-and then SLSA is basically like attach provenance of the artifact, cryptographically sign it, and then check it on any other internal pipeline that you have, okay? So it's a way of doing like zero-trust pipelines that's very effective. It's very good in this current environment. A lot of these attacks are basically pipe-- different kinds of pipeline attacks

messing with your, um, your, your, your tags, your download pages, um, your deps, uh, anything earlier between source and where your users get stuff. Um, libraries is basically, uh, we take-- And this is a new mechanism, so it's a little hard to-- it's a little hard sometimes to-- for people to be like, "Wait, wh-what's the point of this?" Basically, we take clean sc-- clean source. Most attacks are after source and then between the source and the distribution of the artifact. So we take the source, we

airlift it out of the maintainer's kind of like, you know, situation with their distribution page and their CI/CD and everything. We airlift it out, a little helicopter. We put it into our system. We build it all. And then we pass it on to you. Okay, so it's built from source. We also do scanning and other kinds of remediation triage, you know, uh, to, you know, beyond that. But, you know, the build from source is a kind of unique mechanism that we do. And then I guess Ubuntu not updating.

Maybe I should be running on Wolfi. And, um, and then C-- our Chainguard Actions, and we're thinking about more in this space now, but basically trying to remediate this, this problem of CI/CD as being this, this one of the main soft targets in the modern, um, environment, okay? So Chainguard-- you can pull-- you can try it out. It's, it's free to try now. Chainguard Actions, um, you can drop them in for GitHub Actions. Um, we'll also be thinking about other ways where, you know, I think, I don't know what-- exactly what I'm allowed to

say. We have Assemble London going on, so you can tune into that this week, okay? Um, this is something I've been working a lot on myself too, is Chainguard Agent Skills. So Chainguard Agent Skills is basically we take upstream third-party skills, um, we harden them in our pipeline, and then we pass them on to you. Uh, so it goes through a whole bunch of deterministic and non-det-- and agentic steps. I actually just released a weird little video with like, you know, like Machinima kind of Minecraft-esque that will kind of explain

how the pipeline works if you wanna check that out on the-- our YouTube channel. But, um, but yeah. So, you know, and a lot of agent skills like f-- uh, my own in, um, recent, uh, analysis is something like five percent of agent skills on upstream repositories are actively malicious. Um, earlier in the year, I think it was like something like twenty percent, though I think that's changed a little bit. And then another like thirty to forty percent have serious security i-i-issues or implications, um, that make them insecure even though they're not intended to be malicious. Um, we also

can, in your own organization, you can upload, um, uh, skills to be hardened and then discovered by folks within your own organization. Um, so it's basically, uh, quick and dirty. You know, uh, twenty minutes is kind of the new lightning talk, right? And I'm trying to cover a lot of ground. Um, hopefully, we can maybe answer a question or two. Um, but we have the Open Source Finance Forign C-- Forum coming up. It's funny, when I just, just now was looking at this slide, and it said Open Source Fiancé Forum. So that sounded kind of in-- fun actually. We should have the Open Source Fiancé Forum. But,

um, but it's the Open Source Finance Forum on November fifth. Um, that's with Dan Lorenc, our CEO, and, uh, leaders who are-- leaders from the Athena Consortium. Uh, check that out. That's the QR code there. And then ho-hopefully it works. And then, um, this is our fi-- if you're-- I know there's a lot of finance folks in the room. This is-- QR code is to our, like, start here if you're in finance page. Um, come do check out our booth in, um, in... It's booth twenty-three. Um, we're behind a big column, so we're really hard to see. So just look

behind all the columns in the big expo room, and you should be able to find us. And then, um, my name's Patrick Smyth. I, uh, this is my LinkedIn, and this is Wolfi. Uh, Wolfi is the smallest octopus in the world. He's really tiny. He's minimal. Being minimal helps with zero days. Uh, and it's-- he's the size of a, of a fingernail. There's not that many videos of him. This is one of the few that are online. But he, you know... I'll leave him up for a second, and then hopefully we have time for just a couple questions. But thank you.