# Built to Scale: Future-Proofing Your Technology Roadmap John Stecher, Scott Worland · AI Engineer NYC 2026 · October 14 [Watch recording](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19345s) [05:22:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19345s) I'm Scott Worland, I'm the CTO of Norm Ai and I'm privileged enough to be sharing the [05:22:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19350s) stage here with John Stecher who's the CTO of Blackstone. [05:22:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19354s) From what I understand, John has basically three full-time jobs, it sounds like. [05:22:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19359s) Yep. [05:22:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19359s) Responsible for all of the technology at Blackstone, also advises on their investments, [05:22:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19365s) and serves as a resource for the portfolio companies for all things technology. [05:22:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19370s) Did I miss anything? [05:22:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19370s) Nope, I hit it all perfectly, Matt. [05:22:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19372s) Cool job. [05:22:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19374s) Yeah, and if you're not familiar with what Blackstone does, [05:22:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19376s) then you probably have wandered into the wrong conference. [05:22:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19379s) So I'm not going to go into that, but you may not know what Norm does. [05:23:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19381s) So really briefly, Norm is an AI company, and we operate in the legal and regulatory space. [05:23:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19386s) We have an enterprise AI platform that's deployed in some of the largest financial institutions in the world, [05:23:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19392s) and then we also have an affiliated law firm, Norm Law, [05:23:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19395s) and our AI platform also supports that. [05:23:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19399s) So since the end of last year, [05:23:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19401s) and certainly all of this year, [05:23:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19403s) everyone's been racing to figure out how to deploy AI agents. [05:23:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19406s) And I'd love to hear how you're thinking about this at Blackstone [05:23:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19409s) and how you're trying to make sure you get this right. [05:23:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19410s) Yeah, I mean, I think the race has definitely been on [05:23:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19413s) as the technology has improved mightily [05:23:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19415s) since really like February, March of last year, [05:23:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19418s) just in terms of context windows, [05:23:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19419s) being able to run stuff for significantly longer amounts of time have like resulted in people being [05:23:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19424s) able to produce just more value and one of the things that like we've been focused on a lot and [05:23:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19428s) i think every company in our portfolio i advise on and work with all the time that they need to [05:23:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19433s) have top of mind for like being successful with ai agents in general is you have to get your data [05:23:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19438s) story right and so like one of the biggest things impediments and tripping points that i see for [05:24:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19443s) companies out there when they want they're all in on ai they want to build as many agents as possible [05:24:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19447s) They really want to kind of extract value. [05:24:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19449s) If they don't have a cohesive data strategy that's moving stuff out of transactional systems [05:24:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19454s) into something that can be accessed and easily managed and recalled from like an agent perspective, [05:24:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19459s) it is going to fail. [05:24:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19461s) And so that's been one of the biggest things that we've been focused on. [05:24:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19464s) You know, as a big asset manager, we have a ton of data, proprietary data, public data, [05:24:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19469s) all woven together. [05:24:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19469s) We have to move that out of systems into something that we can really leverage from an AI perspective. [05:24:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19475s) I think the second thing that has nothing to do with AI, but you can leverage AI to actually figure out how to do the best possible way, is straight through processing still matters. [05:24:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19484s) I'm always shocked at how everybody just talks about agents as being the biggest differentiator in the world. [05:24:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19490s) The thing that I've leveraged myself using Factory and a couple other ways to go off and build code inside of the firm is taking time, having agents help to craft something for me, understand the workflows out there that people are doing, [05:25:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19502s) and then make sure that I'm actually writing code to optimize for straight-through processing [05:25:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19506s) and then using agents to inject in intelligence where necessary, [05:25:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19510s) where humans were making decisions in the past for a variety of rote work things. [05:25:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19514s) So that's really the two key elements that I spend a ton of time thinking about every single day. [05:25:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19518s) And then the biggest thing from an agent perspective is really stepping back [05:25:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19522s) and looking at what is work that we want to build agents to do better, [05:25:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19527s) to offload intellectual cycles [05:25:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19530s) from our investment professionals, [05:25:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19532s) our finance professionals, our engineers, [05:25:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19534s) such that they can do higher value work. [05:25:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19536s) And there's a ton of stuff [05:25:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19536s) when you think about the software development life cycle, [05:25:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19539s) writing test cases, functional test cases, [05:25:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19541s) performance test cases, [05:25:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19543s) versioning forward libraries underneath applications. [05:25:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19546s) It's the bane of every engineer's existence. [05:25:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19548s) I've hated it for 27 years that I've been doing this. [05:25:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19551s) Having an agent be able to go off [05:25:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19552s) and do these things for me [05:25:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19553s) and renovate my code and my source code [05:25:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19554s) has made all the difference in the world [05:25:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19556s) for the velocity that we deliver at. [05:25:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19558s) So how are you thinking about it at Norm? [05:26:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19560s) Yeah, so Norm was founded in 2023. [05:26:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19563s) So, you know, that's three years, [05:26:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19565s) I guess four years after GPT-1, [05:26:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19567s) but still a year before there was like, [05:26:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19569s) you know, chain of thought. [05:26:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19570s) So it was like an interesting time [05:26:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19571s) in terms of development. [05:26:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19572s) I remember when we had our first team get together, [05:26:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19574s) we were at Stanford's campus for a week. [05:26:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19576s) And I mean, I'm probably misremembering [05:26:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19578s) the exact timeline, [05:26:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19579s) but it felt like we went from 4K to 8K [05:26:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19582s) to 128K context windows. [05:26:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19584s) And then we had multimodal [05:26:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19585s) and everything felt like it was coming all at once. [05:26:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19588s) And then there was a bit of a time where things did slow down when it was coming from the [05:26:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19591s) frontier. [05:26:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19592s) I think like a lot of companies in early 2024, we were trying to deploy these models as part [05:26:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19597s) of our workflow and as part of our production software. [05:26:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19600s) We ended up adding a lot of structure to the system and this comes from all different parts [05:26:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19606s) of the stack. [05:26:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19607s) Whether that was from how we went and took the content that we knew needed to be eventually [05:26:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19612s) added to context to the models and we had very specific ways in which we did that. Or [05:26:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19617s) if there was the sequencing of the models, if there was steps we needed to do in terms [05:27:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19622s) of looking for consistency and then ultimately distilling everything down into some useful [05:27:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19626s) output, we ended up with something that looked like a large map reduce problem effectively. [05:27:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19630s) We had a lot of scaffolding for the agents and we were very particular about how these [05:27:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19636s) models ran. It was much more like a workflow. What this allowed us to do was get really [05:27:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19642s) high quality and also very accurate, but also just very consistent, which is very important [05:27:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19647s) for us in the regulatory space, outputs. [05:27:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19650s) But then one fortuitous thing as part of doing this, because we wanted to have some way to [05:27:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19654s) regression test this big system, we ended up with a really large set of evaluation samples. [05:27:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19659s) We had tens of thousands, actually, of hand-labeled examples by experts. [05:27:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19662s) And so as soon as we moved to a place where it was clear that agents were actually a thing, [05:27:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19667s) and by agents I simply mean, you know, given like an LLM, a system, or given a set of tools [05:27:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19673s) that it can use, give it some goal and give it some way to verify itself against that [05:27:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19677s) goal, allowing it just to go figure out how to do the thing in the best way possible. [05:28:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19680s) We were able to just take this benchmark set and just immediately run on top of it with [05:28:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19685s) some of these more agentic approaches. [05:28:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19686s) And we saw, you know, I think a lot of people had this moment where you're staring at hundreds [05:28:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19690s) of thousands of lines of code your team had really thoughtfully written and realizing [05:28:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19693s) most of it was pretty obsolete and was just kind of actually in the way. [05:28:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19696s) And so to update quickly on this, what we did was we took, I got the engineering leadership team. [05:28:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19702s) We actually rented a Sprinter van and we drove to upstate New York. [05:28:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19706s) It was actually after a big snowstorm. [05:28:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19707s) We had an Airbnb and it was like, forget norm for a minute. [05:28:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19710s) Let's just say what would we build today as far as a platform, knowing what we know right now, if we were going to be starting over. [05:28:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19717s) And then from that, we left with basically a set of specs about like what this platform needed to be able to do to enable two things. [05:28:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19723s) one, deliver agentic experiences to our clients, [05:28:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19726s) and then also it needed to be amenable [05:28:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19728s) to having AI as co-developers [05:28:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19731s) alongside of our own development team. [05:28:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19732s) And that's what we've been executing against ever since then. [05:28:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19735s) So I think this... [05:28:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19737s) Do you have a question? [05:28:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19737s) No, I was going to say, [05:28:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19738s) I think you hit on a key point, Scott, [05:29:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19740s) where it says the eval side of the fence [05:29:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19742s) that you were talking about before, [05:29:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19743s) if you're working in financial services, [05:29:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19745s) the quality bar for what agents have to deliver on [05:29:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19749s) is very, very high. [05:29:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19750s) And so I think really thinking through [05:29:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19752s) like what is your eval set for how you basically test your agents, whatever work they're performing, [05:29:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19759s) that the output makes sense, and it's consistent and repeatable. And I've seen across the board, [05:29:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19763s) as model versions change underneath the covers, you get different answers. You get skew in terms [05:29:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19768s) of the accuracy of information that comes back. So having good evals so that you can run those [05:29:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19773s) consistently and really measure the health of a system is paramount to success. Same thing happens [05:29:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19777s) if harnesses change that you're running stuff on top of. You'll get different answers. Data change [05:29:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19781s) different answers. So it's something where I think if you're trafficking in financial services, [05:29:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19786s) like the duty to be accurate is like up here and hitting that is like something like we think about [05:29:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19792s) all the time. Definitely. I think this leads nicely into this concept that we've been talking [05:29:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19796s) about just like with the bitter lesson. So like everyone's familiar with it [05:30:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19800s) There's this idea that these handcrafted systems built by clever people thinking hard about the problem will oftentimes eventually be dominated by systems that scale with compute. [05:30:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19812s) I think Rich Sutton originally talked about this in 2019, but this has been since the beginning of neural networks. [05:30:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19817s) If you go back to the 50s and 60s, like with Rosenblatt's little perceptron, I think the New York Times ran an article or somebody did saying [05:30:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19823s) It was the first machine that could ever have its own original thoughts. [05:30:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19827s) And there's actually quite a fascinating history here of how the expert systems and the connectionists and what ended up happening. [05:30:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19834s) A lot of it actually was highly litigated in England, and a lot of AI research was dismantled. [05:30:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19839s) But one funny thing, if you go back to the 80s even, there was a lead at IBM's translation lab, and he made a comment, which is very much the bitter lesson. [05:30:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19847s) Every time I fire a linguist, my models improve. [05:30:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19849s) The point of having these linguists in there and trying to think about the rules of language [05:30:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19852s) versus training a statistical model based on probabilities. [05:30:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19855s) We speed up to today and we see this all the time. [05:30:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19858s) We've been touching on lots of points about this. [05:31:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19860s) I'm curious, when you think about your roadmaps, when people on your team are bringing you [05:31:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19863s) project ideas, how do you keep this in mind as you're thinking about this for Blackstone? [05:31:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19868s) A lot of what we do is like the simple equation is giving enough compute. [05:31:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19873s) Every problem is solvable by hardware and software at some point in time in the future. [05:31:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19878s) It's Elon talks about this energy wise, et cetera. [05:31:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19880s) So it's like, it's a big map. [05:31:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19882s) It's a whole bunch of stuff. [05:31:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19883s) How do you bring it back to reality? [05:31:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19885s) Like what I spend a lot of my time doing from like my seat, working with engineers and my [05:31:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19889s) team with product people, with our SRE function, with like stakeholders is really trying to [05:31:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19894s) define like, you know, what is the ROI of what I'm being asked to work on? [05:31:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19898s) And so like when I think of building agents, I think about, you know, getting data into [05:31:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19902s) the right spot, think about building dashboards, trading platforms, whatever it is. [05:31:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19905s) it's like if I look at like what's the ROI that's going to be created from the actual ask that's [05:31:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19911s) coming in to me from a team perspective and then like what's the what's the true cost of actually [05:31:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19916s) building something today like how can I balance that equation and then I make a decision of like [05:32:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19921s) you know is this something where the models are going to get good enough I'm going to have more [05:32:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19925s) than enough compute in six months that it's not worth trying to tackle the problem today [05:32:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19929s) it's better to delay it and actually address it with one of the new models we have made that [05:32:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19934s) decision before is like let's actually wait for the model to become slightly more sophisticated [05:32:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19938s) so that we can solve these problems with it because it will cost less at that time and the [05:32:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19943s) need for that application to be built today isn't pressing there's no like real business value [05:32:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19949s) increase that can be like will be created because it's produced today versus six months from now so [05:32:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19953s) that's a trade-off that we actually you know make regularly and i think about regularly um the other [05:32:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19959s) thing that we do is like you don't want to wait too long to basically build something and so you [05:32:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19963s) can always be waiting for the new model, waiting for the new technology or whatever else. [05:32:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19966s) The beautiful thing with all of the various different software development harnesses today [05:32:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19971s) is if you lean into them, it's amazing how quick you can go from concept, mapping out your planning [05:32:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19977s) and concept with Fable for just, hey, work with me and let's define this with the product person [05:33:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19982s) or with the stakeholder, to actually writing code, to having a workable proof of concept. [05:33:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19987s) And when I say workable proof of concept, it's something people could click through, interact [05:33:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19990s) with whatever else it is, doesn't have cybersecurity review, doesn't have all the other things that [05:33:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19994s) you need to do in our industry to make sure you put something into production that scales, [05:33:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=19998s) but you can get to a POC or a POV very quickly. And so that's one big thing I try to encourage [05:33:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20004s) all my engineers to do frequently is just get to POC. Don't worry about PowerPoints. Don't [05:33:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20009s) worry about anything else. Leverage tokens, which I think it's a fallacy to be like counting tokens [05:33:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20013s) in some way, shape, or form. It's better to think about producing value. Go off produce value, [05:33:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20017s) produce POVs, put it in front of people, iterate on it quickly, and throw it away if you don't need [05:33:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20022s) it. You will spend far less time and money doing those types of things. And so that's like a big [05:33:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20026s) way that we think about trying to solve it. It doesn't maps into the paradigm, like the, you [05:33:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20031s) know, how do you deal with increasing compute, making it easier to do these things. But like, [05:33:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20035s) I just, I think in a much shorter timeframe. Yeah. It's so interesting. Cause it's like, [05:34:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20040s) you know, I mean, you touched on so many things that I think everybody's probably been thinking [05:34:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20044s) about too. And if you go back to 2023, when we were originally like sitting around as a small [05:34:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20048s) group talking about what we're building, we were aware of the bitter lesson then. We talked about [05:34:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20051s) it. We said, how do we not get in this trap? And I think at this point, it's just really evident to [05:34:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20055s) me that it's like, it's definitely a gradient, right? It's not this binary thing about you can [05:34:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20059s) somehow build the perfect technology to index into all the, you know, everything coming in from the [05:34:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20063s) frontier in a way that you don't ever have to throw things away. So really it gets down to like, [05:34:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20067s) where do we want to invest heavily and where are we willing to invest conservatively? Because like [05:34:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20072s) you said, we need something today. But at the same time, we're just aware that some of these [05:34:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20076s) investments we make, particularly around things like we've built fairly complicated systems around [05:34:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20081s) all of our agent primitives. So the harness, the sandbox, how egress works, all of these things. [05:34:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20086s) And we also want to make sure that as things are coming out either from a frontier lab or from a [05:34:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20091s) third party, that we're able to index into that. And so I think a lot of it is just kind of when [05:34:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20096s) do you update? And that's one of the hardest things to do. And for me, a good example was [05:35:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20100s) I updated really hard on Claude Code. [05:35:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20103s) Not so much that it was suddenly tools in my profession [05:35:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20106s) that we could use that were really incredible. [05:35:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20108s) What it really was for me was this idea, [05:35:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20110s) because I was a bit skeptical up to that point [05:35:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20112s) of how we would get something that was truly generalizable. [05:35:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20114s) It's like, well, what does Claude Code give you? [05:35:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20116s) Well, it can write code, and then it can execute the code, [05:35:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20119s) and basically run a computer in some verifiable way. [05:35:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20122s) At that moment, I was like, well, we have all the ingredients [05:35:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20124s) for a truly generalizable intelligence, [05:35:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20126s) because if you can go do this at scale, [05:35:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20127s) then, I mean, you know, of course, like you can basically do anything in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today. [05:35:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20134s) And you fast forward to now and like all these tools that people are using is like AI assistants and things. [05:35:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20139s) This is why they're so amazing is because you can ask a question it's never seen before, fully out of distribution. [05:35:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20143s) It can spin up a computer and go do this thing in the background. [05:35:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20146s) And for a consumer, it's amazing. And we're also seeing this now within the enterprise, too. [05:35:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20150s) And so for me, that moment was like, well, where are we going to invest heavily? [05:35:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20153s) Of course, we still need to invest in what we're building on the agent side. [05:35:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20156s) But going back to what John mentioned, how we do our data capture and distillation, [05:36:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20161s) and how we make this available as state to these models, [05:36:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20163s) and then also just how we do our benchmarks and evaluation. [05:36:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20167s) And so with the law firm, for example, we have this really unique situation [05:36:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20170s) where we started this law firm from nothing. [05:36:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20174s) So we've kind of grown together. [05:36:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20175s) They've grown with Norm together. [05:36:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20176s) And so we work directly with lawyers to stand up these benchmarks and evaluations. [05:36:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20180s) and effectively, rather than think of a traditional roadmap, [05:36:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20184s) like when are you going to build an agent that can go do these three things, [05:36:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20187s) let's have a scorecard that determines when an agent, [05:36:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20190s) or when a human even, is successful at completing these tasks [05:36:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20194s) across various dimensions. [05:36:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20195s) And then we want to be able to see this just improve over time. [05:36:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20198s) So as we make general improvements to the system, [05:36:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20199s) we should see an uplift to these things like an attorney would expect [05:36:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20203s) to improve their workflow, but at the same time, [05:36:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20206s) not getting kind of going back to the state that we were originally, [05:36:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20209s) where we had added a lot of structure to it [05:36:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20211s) in a way that ultimately we found [05:36:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20213s) that we actually could get even better performance [05:36:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20215s) if we were to remove that upfront structure. [05:36:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20217s) I think another thing that's important to think through [05:36:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20219s) is you can produce, like I was saying, [05:37:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20222s) a lot more proof of concepts, proof of value very quickly. [05:37:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20226s) I still think the hard yards for a lot of development out there [05:37:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20229s) is getting it to production. [05:37:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20230s) And what I mean by that is infrastructure build-out [05:37:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20233s) that actually runs inside of the infrastructure [05:37:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20235s) that's available inside of any financial services [05:37:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20237s) or enterprise company, for that matter, out there. [05:37:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20240s) I think that is a challenge that a lot of folks underestimate. [05:37:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20243s) If you're selling AI tools, that is hard on the other side, [05:37:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20247s) just so you know. [05:37:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20247s) Deal with it quite a bit. [05:37:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20249s) The other thing is just dealing with the cybersecurity side of the fence [05:37:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20253s) and the supportability side of the fence. [05:37:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20255s) And so one thing I've noticed, [05:37:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20257s) the agents are getting better at this now, [05:37:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20259s) but it's making sure that the actual software that's being written [05:37:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20262s) is truly traceable, and it feeds into your SRE platform [05:37:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20266s) so that it can be monitorable in production. [05:37:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20268s) Those are things still that I like to have people think about. [05:37:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20271s) It's like they haven't started to scale yet with agents. [05:37:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20273s) They will, but it's something that, from my perspective, [05:37:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20276s) I think about regularly. [05:37:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20278s) Makes sense, and we are approached to this, [05:38:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20280s) and I'm sure a lot of other companies, [05:38:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20281s) and maybe even you guys have taken this approach as well. [05:38:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20284s) There's parts of our code base that we still require [05:38:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20288s) complete human oversight from when it comes to designing, [05:38:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20291s) building, code ownership, [05:38:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20293s) even approving any kind of changes to the code base, [05:38:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20295s) and these are our core primitives and these are things where basically you can build things [05:38:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20300s) on top of these but the things you build on top of these are not allowed to change these [05:38:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20303s) systems without having human review. [05:38:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20305s) It's because of the things you mentioned. [05:38:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20307s) Data security for us is as paramount as delivering quick, it's actually more important. [05:38:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20313s) For where you are and whether it's a consumer product or for us with obviously working with [05:38:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20317s) highly sensitive data, we have to build in so much infrastructure around this and ensure [05:38:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20322s) that we're not ever getting out ahead of these parts of the stack, which it can be hard because [05:38:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20327s) maybe somebody brings to you, and I'm sure you've seen this, a really cool demo and it's [05:38:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20330s) really exciting and you're looking at it terrified. [05:38:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20332s) So this is very cool, but to get to this point in terms of capabilities, it would expose [05:38:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20337s) so much that we're not able to do it. [05:38:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20338s) So I think it's just always that tradeoff. [05:39:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20340s) So we just basically invest heavily in these things and then invest conservatively in places [05:39:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20345s) where we realize as the frontier continues to improve, we want to make sure we have [05:39:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20350s) conduits. The way I described it [05:39:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20351s) actually this morning with the team is like basically [05:39:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20354s) throughout our platform we want to have like conduits [05:39:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20356s) for what's coming from the frontier. You don't want to have [05:39:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20358s) dams where like the intelligence is coming [05:39:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20360s) in and it can't quite get into your stack because you [05:39:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20362s) built things in such a way. Again, it's a great idea. [05:39:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20364s) We're going to get some of it wrong, some of it right. Just being very [05:39:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20366s) cognizant of it, talking about it a lot [05:39:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20367s) and making sure that our core pieces [05:39:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20370s) are locked down but then also [05:39:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20371s) being very open about just [05:39:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20373s) rip and replace. I think that's like a thing [05:39:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20375s) that I think everybody's gotten really good at. [05:39:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20377s) I think that's like one of the biggest [05:39:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20379s) I mean, I've been doing this for 27 years. [05:39:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20381s) Sometimes I feel like it's been like 100 years. [05:39:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20383s) Other times I feel like it's been five. [05:39:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20385s) I think that the thing that's like really made a material difference, [05:39:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20389s) like when I look at the ability to write software today [05:39:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20391s) versus what I was doing 10 years ago, [05:39:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20394s) was 10 years ago it took so much time, effort, and money [05:39:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20397s) in people's intellectual capital [05:40:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20400s) to produce a piece of software. It was almost like the ship had sailed. By the time you designed it and started writing code, it was out to sea and you were going to ride that baby for as long as you could until it got to the other side of the ocean. [05:40:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20410s) and like what I've become very comfortable with today is like for the most part a lot of like [05:40:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20416s) internal applications can be just thrown away it's build them have them work for a purpose and a point [05:40:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20422s) in time but understand that they're going to disappear at some point in time now much more [05:40:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20427s) quickly than ever before and then just think about like what comes next and so I think one of the [05:40:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20432s) challenges with like sitting in my seat or any like good engineer seat should be constantly [05:40:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20436s) questioning, should I rewrite this? And does it cost me less to rewrite it than it costs me to [05:40:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20442s) keep versioning the current version forward? And that is a question that a decade ago never would [05:40:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20447s) have came up in anyone's head if you went to say, I want to rewrite an internal application. [05:40:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20451s) Your CFO would have looked at you like you were insane. And I think now it's a completely relevant [05:40:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20455s) thing that all of us should be doing because net new applications might produce way more business [05:41:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20460s) value at the end of the day. So it's just something I encourage folks to do. It sounds like you guys [05:41:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20463s) are doing quite a bit at Norm. [05:41:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20465s) Yeah, exactly. [05:41:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20466s) And I think it's just that concept [05:41:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20467s) of what used to be expensive [05:41:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20468s) that's now cheap. [05:41:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20469s) And if you don't update quickly on that, [05:41:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20471s) then it's incredible. [05:41:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20473s) You said it feels like 27 or 5 years [05:41:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20475s) or 100 years or 5 years. [05:41:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20476s) It's like if you overindex too hard [05:41:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20479s) on something that was in June [05:41:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20480s) and now it's October, [05:41:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20481s) it feels like in some ways [05:41:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20483s) there's been 10 years [05:41:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20484s) of technological advancement. [05:41:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20486s) Well, great. [05:41:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20486s) We only have a minute or so left. [05:41:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20488s) Is there any closing remarks? [05:41:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20489s) I think the one thing [05:41:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20491s) I'd leave the audience on [05:41:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20492s) if you want good takeaways from this. [05:41:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20496s) We talked about a lot of very interesting things. [05:41:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20498s) What makes me excited to go to work [05:41:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20501s) is it's cool to get back to engineering. [05:41:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20503s) One of the things that I value is [05:41:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20506s) I was a hardcore low-level engineer for 10 years at IBM [05:41:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20509s) and wrote, this was way back in the day, [05:41:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20511s) of using Notepad and VI to write instruction-level code [05:41:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20515s) for RISC machines. [05:41:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20516s) It was awesome. [05:41:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20517s) It's nerdy as all heck. [05:41:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20518s) No one does this stuff anymore, [05:41:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20519s) but it was fun for me. [05:42:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20520s) I still enjoy those days. [05:42:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20521s) as you get more senior in your career and like the barrier to actually writing code on a daily basis [05:42:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20528s) disappears or it becomes more like hard to get through like dealing with a stream like processing [05:42:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20533s) stuff off a stream it's just all these things are like oh my god i can't even get started because [05:42:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20537s) the scaffolding is so hard with like ai it's been like reinvigorating to get back and actually build [05:42:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20543s) stuff because it's like i never lost my knowledge on how to architect things i see this with a lot [05:42:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20547s) of engineers out there, they understand because they've been in the market 10, 20, 30 years, [05:42:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20552s) they understand the complexity of system design, which hasn't changed. [05:42:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20556s) And now they can actually express that knowledge with these new cool agents. [05:42:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20559s) And so I think it's a super exciting time to be like starting a company, working at [05:42:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20562s) a company and like building software because there's like a whole new world that we can [05:42:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20567s) create out there. [05:42:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20567s) So that's what like excites me. [05:42:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20569s) Yeah. [05:42:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20569s) Well, thank you so much. [05:42:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20570s) It's been an honor. [05:42:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20571s) Yep. [05:42:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20571s) Thank you guys for having us here today. [05:42:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20572s) I appreciate it. [05:42:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20573s) Thank you.