[
  {
    "startMs": 12900,
    "text": "I'm pretty excited to be here and talk about who verifies the software factory. Uh, before we get started, uh, can you give me a quick show of hands, uh, raise your hand if your coding agent has pulled, uh, opened any pull request in the last one week. Almost everyone. Thanks. And keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged.  Oh, I do see a few people. That's excellent. So what I'm going"
  },
  {
    "startMs": 42880,
    "text": "to talk about is the gap between the, uh, two, uh, camps, uh, merging requests without reviewing manually and, uh, you know, merging requests, uh, after doing some reviews. So my name is Manish Kapur, and I lead the product and solut-- uh, developer relations, uh, team at Sonar, and I'm excited to be here, uh, with my guest speaker, Addy Osmani from Anthropic. Uh, Addy has been, uh, had been at Google"
  },
  {
    "startMs": 72880,
    "text": "for about fourteen years. Most of you probably know him, and he's written lots and lots of content and articles. Uh, I learn a lot from Addy, so I'm, uh, really fortunate to have him here stand next to me and, uh, share his wisdom with all of us here today. Uh, he recently joined Anthropic. And Addy, so congratulations on your, uh, role at-"
  },
  {
    "startMs": 92600,
    "text": "Thanks"
  },
  {
    "startMs": 92660,
    "text": "... uh, Anthropic. Would you like to tell the team what you're working on at Anthropic, what your role is?"
  },
  {
    "startMs": 97140,
    "text": "Yeah, sure. So I'm responsible for making sure that y'all have a really great developer experience building with Claude Code. And any time that we discover there are opportunities to make it even smoother, we bring that right back into the product. We ship a couple of times every single day, and so we're very interested in trying to make sure that you have the best possible experience as often as possible."
  },
  {
    "startMs": 116180,
    "text": "Cool. Thank you. All right. Uh, s-so let's start with some problem and context setting slide. Uh, there are two numbers on the slide. CircleCI analyzed twenty-eight million CI workflows, uh, this year, and they noticed that the throughput jumped by about fifty-nine percent, uh, but yet three in ten attempts to merge that code into production are failing. And this is a stat from one, uh, independent stat from, uh, CircleCI. Uh, and I think I've seen the"
  },
  {
    "startMs": 146120,
    "text": "same trend for most of our customers that I speak to every day. Uh, this is a quote from Addy: \"A software factory is harnessing loops at scale.\" This resonated quite a bit for me. And if you've not read that article, you can look it up. It's p-pretty, uh, uh, educational article and talking about f-future, how the future will look like in terms of soft- uh, software factories. Uh, so Addy, would you like to expand on what you were thinking when you wrote this article about soft loops-"
  },
  {
    "startMs": 173640,
    "text": "Yeah"
  },
  {
    "startMs": 173700,
    "text": "... and harnessing the loops at scale?"
  },
  {
    "startMs": 175420,
    "text": "Yeah, absolutely. So a lot of folks here have probably heard this term software factories, and there's two types of factories that are at least worth keeping in mind. W-- You know, software factories harnessing loops at scale. A lights-on factory is one where you have agents that are helping you, um, with planning, building, checking, and shipping code again and again, but you as a person still apply your judgment and taste inside the picture. And then we have a dark software factory, which is a little bit closer to how, you know, a car factory with"
  },
  {
    "startMs": 205480,
    "text": "robotics might be working, where it's constantly doing the work, so it's scoping, building, checking, and you as the person move down, um, further to thinking about intent and you're pulled in when they're really the greatest of exceptions to the case. But that's sort of what I was thinking. I think a lot of teams are somewhere on this spectrum, um, and are on this journey to figure out, like, how, how do I get further and further into automation in a way that I can trust it?"
  },
  {
    "startMs": 230180,
    "text": "Cool. So, uh, AI coding is now basically a solved problem. We all know, uh, agents are there. Eighty-five percent of the organizations are already using coding agents. It'll go to ninety-five percent and very soon to hundred percent. Uh, seventy-five percent of the, uh, dev-- seventy-six percent of the developers that generate AI code, uh, they are saying that it's being, uh, code is being produced at a speed that is impossible for them to verify it. So they are spending more and more time reviewing the code. So what do you think is the most pressing question"
  },
  {
    "startMs": 260279,
    "text": "that engineering leaders have right now?"
  },
  {
    "startMs": 263340,
    "text": "So I think that teams have gotten better at working with agents over the last year. Um, models, harnesses have also gotten better. And what I see a lot of leaders being interested in is how do we enable you to ship code faster, um, with confidence? And that confidence piece is very important. Agents can now generate ways for you to check your code much more cheaply than ever before. And if you think about the things that enabled us to call good code, you know, high quality, uh, those were things like good"
  },
  {
    "startMs": 293340,
    "text": "tests, benchmarks, uh, specs, anything that gives us confidence. So agents are able to help us generate that a little bit more easily these days, a little bit more quickly. We can, of course, refine those things, but I think that those checks are one of the most important things we should be spending our time on these days. Like, if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct. If you've got that in place and y-you can make sure that the code is both correct"
  },
  {
    "startMs": 323200,
    "text": "and complete, that can easily two to three by, like, the quality of your outcomes."
  },
  {
    "startMs": 327880,
    "text": "Yep. Yep, absolutely right. Spot on. I think code confidence, uh, we have been running an ad, code confidence is sc-scarce, and you really need to build it up now that the coding problem is solved. So I, uh, actually I, yesterday night, so this might be a surprise question for you. Uh, yesterday night, I read a post, uh, from Anthropic, and I encourage all of you to read it. It talks about the scale at which Anthropic is operating. Uh, they are merging about thirty thousand code changes every single day. They used to do about three thousand or so, twenty-five"
  },
  {
    "startMs": 357840,
    "text": "hundred or so code changes just a few months back. They have scaled from 2,500 code changes to 3,000-- uh, 30,000 code changes per day. That is mind-boggling. So Addy, what does code review look like at, uh, Claude Code at, at Anthropic and with the Claude Code team that you're working with?"
  },
  {
    "startMs": 373404,
    "text": "Yeah, these are great questions. So we were s- we were merging about 2,500 changes all the way back in May. Um, and even for our recent refresh of claude.ai, we put out a case study saying that we kind of 3x'd the performance of it. That was accomplished by merging about 3,000 pull requests in a very short space of time with engineers, product designers, a number of different functions contributing to it. So our workflow for being able to get verification and code review in place is that we first of all have Claude do a first-pass"
  },
  {
    "startMs": 403484,
    "text": "review. What's very important in there is making sure that it has a way to actually try the experience out, similar to a real user. So that can mean end-to-end tests, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence. Um, people usually ask us, \"Okay, well, what happens if, you know, you discover a bug or a gap in functionality?\" Well, we have hooks that will then go and run on any edits. They will study the diffs and allow"
  },
  {
    "startMs": 433404,
    "text": "us to get this loop in place where we're correcting the PR as quickly as possible. So we have, uh, Claude Code doing the review. We have a, um, code review product, uh, that you can also check out that's pretty good, and the idea there is we're trying to automate code review, um, as much as possible. This is something that we use ourselves. Um, and increasingly what's important is for your teams to be thinking about how can you cod-- We talk about taste and judgment so much these days-"
  },
  {
    "startMs": 459624,
    "text": "Yes"
  },
  {
    "startMs": 459644,
    "text": "... right? It's important for you to be trying to codify what you consider best practice and taste in your team's shared, like CLAUDE.md, AGENTS.md, anywhere where the agent can then consult during code review. And then people are responsible for checking on the agent's evidence. So how do we know? Some-- If I'm building out, um, a piece of UI that's gonna be, you know, used by a user in the browser, I wanna make sure that it's tested in the browser, maybe generated me a video of that experience where it's been clicking around"
  },
  {
    "startMs": 489584,
    "text": "and gives me confidence, well, okay, well, this works. If I care about the mobile or desktop experience, maybe it generates me screenshots or videos of that. But basically you wanna get to a place where the agent's giving you as much verification evidence as possible."
  },
  {
    "startMs": 501364,
    "text": "Cool. Yeah, verification is key. I think you need to verify everything that the, uh, software factories will produce, and that's what we are going to talk about today. Uh, so, uh, another s- uh, these stats, by the way, are from, uh, the State of Code survey report that will be coming out very soon from Sonar. Uh, we surveyed about a thousand-- more than thousand developers, and we found some interesting data which will give us insights as to what's happening in the industry today and where we are headed. So 84% say AI features, uh, uh, gets, uh, gets features and fixes"
  },
  {
    "startMs": 531224,
    "text": "p-to production faster. The pull requests are getting larger and more complex. Uh, and, you know, yet, uh, only th- uh, 31% more code is reaching, reaching the production without review. So that's a little, little bit of a scary fact, particularly for, uh, production, uh, for applications in the regulated industries, in the mission critical applications where you absolutely ma- need to make sure, uh, that, you know, uh, you have to verify that code and you're confident in the code that is being shipped. So, uh, what are"
  },
  {
    "startMs": 561224,
    "text": "the-- where are the next gains going to come from, Addy?"
  },
  {
    "startMs": 563964,
    "text": "I think the next gains are gonna come from a combination of us really nailing verification, right? As well as model quality. But models are already, like if you take a look at Opus 5.5, model quality has already gotten to a place where it's pretty good. Um, and I would say that us being able to share best practices about that verification step is gonna become increasingly more important. Because if you tell people, \"Yeah, just like verify the work,\" you know, their next question is, \"What does that even mean?\""
  },
  {
    "startMs": 587484,
    "text": "Yeah."
  },
  {
    "startMs": 587704,
    "text": "Right?"
  },
  {
    "startMs": 588084,
    "text": "Yeah."
  },
  {
    "startMs": 588244,
    "text": "And so I think that the conversation needs to evolve into specifically what kinds of tests are useful. We lean a lot on end-to-end tests, property tests, any visual verification tests, things that can help us just get that confidence to merge those changes in automatically."
  },
  {
    "startMs": 602404,
    "text": "Cool. You mentioned light factories wanna go dark, and that's the way-- uh, direction we are headed. Uh, our survey results show that, you know, today 5% of the co- companies are using, uh, software factories. It will get to 34%, and surprisingly 39% in the financial services. How many of you are using software factories today? Just curious."
  },
  {
    "startMs": 622564,
    "text": "A handful of you. And how many are planning to use software factories by next year?"
  },
  {
    "startMs": 627024,
    "text": "How do you define a software factory?"
  },
  {
    "startMs": 629164,
    "text": "Yeah, that's a good question. How do you define a software factory? Uh, would you like to take that question or should-"
  },
  {
    "startMs": 633004,
    "text": "Yeah, yeah, sure. So we, we talked about, uh, definition of software factories just a little bit earlier. A software factory is basically loops at scale, and when we think about this, it's all about, like, what end of the spectrum are you at in terms of how much people are still involved in the process? Um, the more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark. And I think that, yeah, most people are still somewhere in that spectrum and are not quite, as this data shows-"
  },
  {
    "startMs": 659124,
    "text": "Yeah"
  },
  {
    "startMs": 659144,
    "text": "... at that dark point just yet."
  },
  {
    "startMs": 660404,
    "text": "Yep. Just to add to what Addy said, I think software factory, think of it like a production line where you start defining the, uh, intent and your specifications, then you start writing the code. So it's an assembly line where each of these software development lifecycle, uh, steps happen, and automating them at scale where there's, uh, pre-pretty much in a dark fa- uh, factory there would be no, uh, lights on. It's autonomously working, fully autonomously working. A light factory has, uh, uh, is where people are involved in the process at"
  },
  {
    "startMs": 690324,
    "text": "every step or most steps. Uh, whereas a dark factory, the goal would be to turn off the lights and let the factory run independently, uh, twenty-four by seven. Uh, all right. So you talked about verification. Uh, I'll talk a little bit about what we think at Sonar about verification. If you saw our keynote earlier today from our CEO, he talked about, uh, verification should be multi-layered. Uh, we absolutely 100% believe that each company should adopt a zero-trust multi-layered verification approach, and verification should be"
  },
  {
    "startMs": 720244,
    "text": "done in every loop. Whether you're operating inside the agentic loop or whether you're working inside the CI verification loop, or whether you're doing code maintenance. In each and every loop, it is important to verify the output of the factories or the agents that, uh, code that the agents are producing. So we look at it in, uh, terms of layers, and, uh, think of it like, you know, the f- this first layer, which is a deterministic layer, which catches, uh, bugs, security issues, things like SQL injection,"
  },
  {
    "startMs": 750732,
    "text": "uh, buffer overflow bugs, and all of those are caught. And then there's a second layer where agentic review happens, where agents are used to like, uh, Addy was talking about, agents review, review that code. And in that agentic review they catch logic and intent issues, which the deterministic, uh, layer is not well-suited to catch. So we, we use each layer is targeted at catching a specific types of issues. And then the third layer could be a software supply chain security layer, where you look at all the CVEs, all your, uh,"
  },
  {
    "startMs": 780432,
    "text": "malware, uh, uh, detection happens, all your, uh, malicious package detection happens. All of that happens in the third layer. So it's a layered approach where each layer is tuned and optimized to catch a certain type of issue. So, uh, we absolutely recommend doing a multi-layered verification. Oops."
  },
  {
    "startMs": 799692,
    "text": "So there's another stat here. 28%, uh, you know, uh, companies have approved agents and shared controls, and very few shared controls. And, uh, 5% are operating, uh, software factories autonomously. What are your thoughts there, Addy, uh, in terms of-"
  },
  {
    "startMs": 817432,
    "text": "So I've talked to so many companies where a couple of months ago, one of their biggest worries was they were allowing their engineering teams to go off and play around with agents, and they got to a place where there were hundreds, thousands of agents in some cases, and they just didn't know what they were doing. Like, what are these agents actually doing? They're touching all of our systems, accessing all kinds of data. Maybe they're deploying to production. And so a lot of enterprises think about concerns like, uh, you know, agent identity and telemetry and being able to track what's actually happening."
  },
  {
    "startMs": 845932,
    "text": "Yeah."
  },
  {
    "startMs": 846112,
    "text": "And I'd say that the three biggest things for a lot of companies are: How do we think about control? How do we think about permissions? And how do we think about the sandbox? Like, how are we making sure that the agent isn't jumping out of the sandbox and doing things that it shouldn't be doing?"
  },
  {
    "startMs": 857292,
    "text": "Yeah. Yeah. We, we, uh, at Sonar, we call it guide rails. So instead of guard rails, we're thinking of calling it as guide rails, where we guide the agents to follow the rail tracks, right? Uh, a specific path, uh, to take and prevent them from, uh, going out of bounds or so to say, going out of the sandbox, like, uh, Addy was saying. So it's very, very important to have guide rails for our agent and govern the agents that you're working with. Uh, so let's, uh, we're coming to a close now, uh, as my timer is telling me we just have a few minutes left."
  },
  {
    "startMs": 887392,
    "text": "Let's go get into a lightning round. Uh, one sentence response for each of these questions. As, and you have seen the questions already on the slide, so-"
  },
  {
    "startMs": 894052,
    "text": "Can these be long sentences?"
  },
  {
    "startMs": 896432,
    "text": "They can be as long as the time permits."
  },
  {
    "startMs": 898732,
    "text": "Perfect."
  },
  {
    "startMs": 899472,
    "text": "Uh, light or dark factory for most teams by end of 2027, yes or no?"
  },
  {
    "startMs": 903732,
    "text": "I think we're gonna see a light factories with some dark corners. I don't think we're going to get to everybody in this room using dark factories entirely, but it's gonna be something close to that."
  },
  {
    "startMs": 912992,
    "text": "I would guess, uh, Anthropic would do it this year or next year."
  },
  {
    "startMs": 920692,
    "text": "All right. Second question, should agent-written code be labeled in the PR?"
  },
  {
    "startMs": 926192,
    "text": "I personally think it should. Historically, we have, uh, felt a great deal of ownership around our code, pride around the fact that, hey, I wrote this thing. I'm putting my name on it. And at Anthropic, like we, we think about this a little bit differently. We've evolved to the point where we don't really mind so much if Claude is like labeled as, \"Hey, I'm, I'm the one that wrote this thing,\" because we're a little bit more focused on the outcomes and making sure that the code, the features, like what we're shipping to users, um, is the right thing. And so I personally think it's fine for the code to be written,"
  },
  {
    "startMs": 956232,
    "text": "you know, labeled as being coming from an agent, and that can also help us with things like figuring out, okay, well, if the agent wrote it, what are some of the constraints and extra checks that we need to keep in place."
  },
  {
    "startMs": 964272,
    "text": "I, I, I have the same point of view, Addy. I think in the short run, at least in the foreseeable, uh, you know, six to nine months, we should label the PRs till we are confident enough in our factories. We do want to know and track it, and it's also important for the financial customers here in terms of auditability. If you, uh, you know, if you, uh, label it, uh, you know who's writing the code-"
  },
  {
    "startMs": 984332,
    "text": "Yeah"
  },
  {
    "startMs": 984332,
    "text": "... because we are, at the end of the day, still accountable for it."
  },
  {
    "startMs": 986332,
    "text": "Yeah, absolutely. Accountability is the most important part here. Like, we still label, like, who is the owner for this change? I might have kicked off a Cloud R- a Claude run to complete a piece of work, but I'm, I'm still on the hook."
  },
  {
    "startMs": 997032,
    "text": "Yep."
  },
  {
    "startMs": 997072,
    "text": "If something goes wrong, I have to go and fix it."
  },
  {
    "startMs": 999132,
    "text": "Yep, yep. Key point. Yep. Uh, the next question, actually, I'll improvise it here because this is a leadership ses- session. O- one skill a junior developer or leaders should, uh, build on this year."
  },
  {
    "startMs": 1011352,
    "text": "I think that you should push your teams... If you're a leader, you should push your teams to really think hard about what verification means. Um, how are you already approaching things like testing? What is the gap between that and you getting to a place where you're very comfortable with the agents being able to sign off on the code being good enough to merge on their own? And so figuring that out. If you're a junior, it's the other side of that, like actually getting more experience in the different flavors of tests, whether it's like end-to-end tests, property tests, et cetera."
  },
  {
    "startMs": 1038652,
    "text": "Yep. And the last, uh, you, you actually not following the rules, you're going beyond one sentence, but that's cool. Uh, the last one-"
  },
  {
    "startMs": 1046952,
    "text": "We have 30 seconds left. I'm fine."
  },
  {
    "startMs": 1048552,
    "text": "We can go a minute over, I'm sure. Uh, the agent opened the PR. It is safe to merge when?"
  },
  {
    "startMs": 1055532,
    "text": "You have full confidence nothing's gonna go wrong or that you've mitigated that-"
  },
  {
    "startMs": 1058452,
    "text": "Love it"
  },
  {
    "startMs": 1058692,
    "text": "... as much as possible."
  },
  {
    "startMs": 1059752,
    "text": "Love it. All right. Uh, here's my last slide, and then we'll wrap it up. Uh, so what separates, uh, teams that are shipping, uh, fast but not shipping blind? I think that three key takeaways for you. Uh, writing code is the fastest part, and that is solved. Strong review and verification for all teams, uh, is what we- will help you maintain that speed. Fast to write is not fast to ship. Uh, you have to basically count what is getting merged, look at what is getting merged, and make sure it doesn't escape the sandbox and the things like"
  },
  {
    "startMs": 1089692,
    "text": "that before it gets into the production. Verify everything. And autonomy is earned. Uh, you-- We will get to software factories very soon, if not next year. Uh, you need to put an independent automated verification inside each loop of your factory. Make sure that you can trust the output that the factory is producing. Uh, with that, I'll wrap it up. Thanks for attending our talk today, uh, a conversation with Addy. Appre- again, Addy, appreciate you taking the time and joining us here in this conversation."
  },
  {
    "startMs": 1117552,
    "text": "Of course. Thanks for having me."
  },
  {
    "startMs": 1118712,
    "text": "Uh, visit us at the big red booth, uh, the Sonar booth in the expo hall if you have any further questions or would like to, uh, talk about it. Thank you again."
  }
]