Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
5:22:25I'm Scott Worland, I'm the CTO of Norm Ai and I'm privileged enough to be sharing the
5:22:30stage here with John Stecher who's the CTO of Blackstone.
5:22:34From what I understand, John has basically three full-time jobs, it sounds like.
5:22:39Yep.
5:22:39Responsible for all of the technology at Blackstone, also advises on their investments,
5:22:45and serves as a resource for the portfolio companies for all things technology.
5:22:50Did I miss anything?
5:22:50Nope, I hit it all perfectly, Matt.
5:22:52Cool job.
5:22:54Yeah, and if you're not familiar with what Blackstone does,
5:22:56then you probably have wandered into the wrong conference.
5:22:59So I'm not going to go into that, but you may not know what Norm does.
5:23:01So really briefly, Norm is an AI company, and we operate in the legal and regulatory space.
5:23:06We have an enterprise AI platform that's deployed in some of the largest financial institutions in the world,
5:23:12and then we also have an affiliated law firm, Norm Law,
5:23:15and our AI platform also supports that.
5:23:19So since the end of last year,
5:23:21and certainly all of this year,
5:23:23everyone's been racing to figure out how to deploy AI agents.
5:23:26And I'd love to hear how you're thinking about this at Blackstone
5:23:29and how you're trying to make sure you get this right.
5:23:30Yeah, I mean, I think the race has definitely been on
5:23:33as the technology has improved mightily
5:23:35since really like February, March of last year,
5:23:38just in terms of context windows,
5:23:39being able to run stuff for significantly longer amounts of time have like resulted in people being
5:23:44able to produce just more value and one of the things that like we've been focused on a lot and
5:23:48i think every company in our portfolio i advise on and work with all the time that they need to
5:23:53have top of mind for like being successful with ai agents in general is you have to get your data
5:23:58story right and so like one of the biggest things impediments and tripping points that i see for
5:24:03companies out there when they want they're all in on ai they want to build as many agents as possible
5:24:07They really want to kind of extract value.
5:24:09If they don't have a cohesive data strategy that's moving stuff out of transactional systems
5:24:14into something that can be accessed and easily managed and recalled from like an agent perspective,
5:24:19it is going to fail.
5:24:21And so that's been one of the biggest things that we've been focused on.
5:24:24You know, as a big asset manager, we have a ton of data, proprietary data, public data,
5:24:29all woven together.
5:24:29We have to move that out of systems into something that we can really leverage from an AI perspective.
5:24:35I think the second thing that has nothing to do with AI, but you can leverage AI to actually figure out how to do the best possible way, is straight through processing still matters.
5:24:44I'm always shocked at how everybody just talks about agents as being the biggest differentiator in the world.
5:24:50The thing that I've leveraged myself using Factory and a couple other ways to go off and build code inside of the firm is taking time, having agents help to craft something for me, understand the workflows out there that people are doing,
5:25:02and then make sure that I'm actually writing code to optimize for straight-through processing
5:25:06and then using agents to inject in intelligence where necessary,
5:25:10where humans were making decisions in the past for a variety of rote work things.
5:25:14So that's really the two key elements that I spend a ton of time thinking about every single day.
5:25:18And then the biggest thing from an agent perspective is really stepping back
5:25:22and looking at what is work that we want to build agents to do better,
5:25:27to offload intellectual cycles
5:25:30from our investment professionals,
5:25:32our finance professionals, our engineers,
5:25:34such that they can do higher value work.
5:25:36And there's a ton of stuff
5:25:36when you think about the software development life cycle,
5:25:39writing test cases, functional test cases,
5:25:41performance test cases,
5:25:43versioning forward libraries underneath applications.
5:25:46It's the bane of every engineer's existence.
5:25:48I've hated it for 27 years that I've been doing this.
5:25:51Having an agent be able to go off
5:25:52and do these things for me
5:25:53and renovate my code and my source code
5:25:54has made all the difference in the world
5:25:56for the velocity that we deliver at.
5:25:58So how are you thinking about it at Norm?
5:26:00Yeah, so Norm was founded in 2023.
5:26:03So, you know, that's three years,
5:26:05I guess four years after GPT-1,
5:26:07but still a year before there was like,
5:26:09you know, chain of thought.
5:26:10So it was like an interesting time
5:26:11in terms of development.
5:26:12I remember when we had our first team get together,
5:26:14we were at Stanford's campus for a week.
5:26:16And I mean, I'm probably misremembering
5:26:18the exact timeline,
5:26:19but it felt like we went from 4K to 8K
5:26:22to 128K context windows.
5:26:24And then we had multimodal
5:26:25and everything felt like it was coming all at once.
5:26:28And then there was a bit of a time where things did slow down when it was coming from the
5:26:31frontier.
5:26:32I think like a lot of companies in early 2024, we were trying to deploy these models as part
5:26:37of our workflow and as part of our production software.
5:26:40We ended up adding a lot of structure to the system and this comes from all different parts
5:26:46of the stack.
5:26:47Whether that was from how we went and took the content that we knew needed to be eventually
5:26:52added to context to the models and we had very specific ways in which we did that. Or
5:26:57if there was the sequencing of the models, if there was steps we needed to do in terms
5:27:02of looking for consistency and then ultimately distilling everything down into some useful
5:27:06output, we ended up with something that looked like a large map reduce problem effectively.
5:27:10We had a lot of scaffolding for the agents and we were very particular about how these
5:27:16models ran. It was much more like a workflow. What this allowed us to do was get really
5:27:22high quality and also very accurate, but also just very consistent, which is very important
5:27:27for us in the regulatory space, outputs.
5:27:30But then one fortuitous thing as part of doing this, because we wanted to have some way to
5:27:34regression test this big system, we ended up with a really large set of evaluation samples.
5:27:39We had tens of thousands, actually, of hand-labeled examples by experts.
5:27:42And so as soon as we moved to a place where it was clear that agents were actually a thing,
5:27:47and by agents I simply mean, you know, given like an LLM, a system, or given a set of tools
5:27:53that it can use, give it some goal and give it some way to verify itself against that
5:27:57goal, allowing it just to go figure out how to do the thing in the best way possible.
5:28:00We were able to just take this benchmark set and just immediately run on top of it with
5:28:05some of these more agentic approaches.
5:28:06And we saw, you know, I think a lot of people had this moment where you're staring at hundreds
5:28:10of thousands of lines of code your team had really thoughtfully written and realizing
5:28:13most of it was pretty obsolete and was just kind of actually in the way.
5:28:16And so to update quickly on this, what we did was we took, I got the engineering leadership team.
5:28:22We actually rented a Sprinter van and we drove to upstate New York.
5:28:26It was actually after a big snowstorm.
5:28:27We had an Airbnb and it was like, forget norm for a minute.
5:28:30Let's just say what would we build today as far as a platform, knowing what we know right now, if we were going to be starting over.
5:28:37And then from that, we left with basically a set of specs about like what this platform needed to be able to do to enable two things.
5:28:43one, deliver agentic experiences to our clients,
5:28:46and then also it needed to be amenable
5:28:48to having AI as co-developers
5:28:51alongside of our own development team.
5:28:52And that's what we've been executing against ever since then.
5:28:55So I think this...
5:28:57Do you have a question?
5:28:57No, I was going to say,
5:28:58I think you hit on a key point, Scott,
5:29:00where it says the eval side of the fence
5:29:02that you were talking about before,
5:29:03if you're working in financial services,
5:29:05the quality bar for what agents have to deliver on
5:29:09is very, very high.
5:29:10And so I think really thinking through
5:29:12like what is your eval set for how you basically test your agents, whatever work they're performing,
5:29:19that the output makes sense, and it's consistent and repeatable. And I've seen across the board,
5:29:23as model versions change underneath the covers, you get different answers. You get skew in terms
5:29:28of the accuracy of information that comes back. So having good evals so that you can run those
5:29:33consistently and really measure the health of a system is paramount to success. Same thing happens
5:29:37if harnesses change that you're running stuff on top of. You'll get different answers. Data change
5:29:41different answers. So it's something where I think if you're trafficking in financial services,
5:29:46like the duty to be accurate is like up here and hitting that is like something like we think about
5:29:52all the time. Definitely. I think this leads nicely into this concept that we've been talking
5:29:56about just like with the bitter lesson. So like everyone's familiar with it
5:30:00There's this idea that these handcrafted systems built by clever people thinking hard about the problem will oftentimes eventually be dominated by systems that scale with compute.
5:30:12I think Rich Sutton originally talked about this in 2019, but this has been since the beginning of neural networks.
5:30:17If you go back to the 50s and 60s, like with Rosenblatt's little perceptron, I think the New York Times ran an article or somebody did saying
5:30:23It was the first machine that could ever have its own original thoughts.
5:30:27And there's actually quite a fascinating history here of how the expert systems and the connectionists and what ended up happening.
5:30:34A lot of it actually was highly litigated in England, and a lot of AI research was dismantled.
5:30:39But one funny thing, if you go back to the 80s even, there was a lead at IBM's translation lab, and he made a comment, which is very much the bitter lesson.
5:30:47Every time I fire a linguist, my models improve.
5:30:49The point of having these linguists in there and trying to think about the rules of language
5:30:52versus training a statistical model based on probabilities.
5:30:55We speed up to today and we see this all the time.
5:30:58We've been touching on lots of points about this.
5:31:00I'm curious, when you think about your roadmaps, when people on your team are bringing you
5:31:03project ideas, how do you keep this in mind as you're thinking about this for Blackstone?
5:31:08A lot of what we do is like the simple equation is giving enough compute.
5:31:13Every problem is solvable by hardware and software at some point in time in the future.
5:31:18It's Elon talks about this energy wise, et cetera.
5:31:20So it's like, it's a big map.
5:31:22It's a whole bunch of stuff.
5:31:23How do you bring it back to reality?
5:31:25Like what I spend a lot of my time doing from like my seat, working with engineers and my
5:31:29team with product people, with our SRE function, with like stakeholders is really trying to
5:31:34define like, you know, what is the ROI of what I'm being asked to work on?
5:31:38And so like when I think of building agents, I think about, you know, getting data into
5:31:42the right spot, think about building dashboards, trading platforms, whatever it is.
5:31:45it's like if I look at like what's the ROI that's going to be created from the actual ask that's
5:31:51coming in to me from a team perspective and then like what's the what's the true cost of actually
5:31:56building something today like how can I balance that equation and then I make a decision of like
5:32:01you know is this something where the models are going to get good enough I'm going to have more
5:32:05than enough compute in six months that it's not worth trying to tackle the problem today
5:32:09it's better to delay it and actually address it with one of the new models we have made that
5:32:14decision before is like let's actually wait for the model to become slightly more sophisticated
5:32:18so that we can solve these problems with it because it will cost less at that time and the
5:32:23need for that application to be built today isn't pressing there's no like real business value
5:32:29increase that can be like will be created because it's produced today versus six months from now so
5:32:33that's a trade-off that we actually you know make regularly and i think about regularly um the other
5:32:39thing that we do is like you don't want to wait too long to basically build something and so you
5:32:43can always be waiting for the new model, waiting for the new technology or whatever else.
5:32:46The beautiful thing with all of the various different software development harnesses today
5:32:51is if you lean into them, it's amazing how quick you can go from concept, mapping out your planning
5:32:57and concept with Fable for just, hey, work with me and let's define this with the product person
5:33:02or with the stakeholder, to actually writing code, to having a workable proof of concept.
5:33:07And when I say workable proof of concept, it's something people could click through, interact
5:33:10with whatever else it is, doesn't have cybersecurity review, doesn't have all the other things that
5:33:14you need to do in our industry to make sure you put something into production that scales,
5:33:18but you can get to a POC or a POV very quickly. And so that's one big thing I try to encourage
5:33:24all my engineers to do frequently is just get to POC. Don't worry about PowerPoints. Don't
5:33:29worry about anything else. Leverage tokens, which I think it's a fallacy to be like counting tokens
5:33:33in some way, shape, or form. It's better to think about producing value. Go off produce value,
5:33:37produce POVs, put it in front of people, iterate on it quickly, and throw it away if you don't need
5:33:42it. You will spend far less time and money doing those types of things. And so that's like a big
5:33:46way that we think about trying to solve it. It doesn't maps into the paradigm, like the, you
5:33:51know, how do you deal with increasing compute, making it easier to do these things. But like,
5:33:55I just, I think in a much shorter timeframe. Yeah. It's so interesting. Cause it's like,
5:34:00you know, I mean, you touched on so many things that I think everybody's probably been thinking
5:34:04about too. And if you go back to 2023, when we were originally like sitting around as a small
5:34:08group talking about what we're building, we were aware of the bitter lesson then. We talked about
5:34:11it. We said, how do we not get in this trap? And I think at this point, it's just really evident to
5:34:15me that it's like, it's definitely a gradient, right? It's not this binary thing about you can
5:34:19somehow build the perfect technology to index into all the, you know, everything coming in from the
5:34:23frontier in a way that you don't ever have to throw things away. So really it gets down to like,
5:34:27where do we want to invest heavily and where are we willing to invest conservatively? Because like
5:34:32you said, we need something today. But at the same time, we're just aware that some of these
5:34:36investments we make, particularly around things like we've built fairly complicated systems around
5:34:41all of our agent primitives. So the harness, the sandbox, how egress works, all of these things.
5:34:46And we also want to make sure that as things are coming out either from a frontier lab or from a
5:34:51third party, that we're able to index into that. And so I think a lot of it is just kind of when
5:34:56do you update? And that's one of the hardest things to do. And for me, a good example was
5:35:00I updated really hard on Claude Code.
5:35:03Not so much that it was suddenly tools in my profession
5:35:06that we could use that were really incredible.
5:35:08What it really was for me was this idea,
5:35:10because I was a bit skeptical up to that point
5:35:12of how we would get something that was truly generalizable.
5:35:14It's like, well, what does Claude Code give you?
5:35:16Well, it can write code, and then it can execute the code,
5:35:19and basically run a computer in some verifiable way.
5:35:22At that moment, I was like, well, we have all the ingredients
5:35:24for a truly generalizable intelligence,
5:35:26because if you can go do this at scale,
5:35:27then, I mean, you know, of course, like you can basically do anything in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today.
5:35:34And you fast forward to now and like all these tools that people are using is like AI assistants and things.
5:35:39This is why they're so amazing is because you can ask a question it's never seen before, fully out of distribution.
5:35:43It can spin up a computer and go do this thing in the background.
5:35:46And for a consumer, it's amazing. And we're also seeing this now within the enterprise, too.
5:35:50And so for me, that moment was like, well, where are we going to invest heavily?
5:35:53Of course, we still need to invest in what we're building on the agent side.
5:35:56But going back to what John mentioned, how we do our data capture and distillation,
5:36:01and how we make this available as state to these models,
5:36:03and then also just how we do our benchmarks and evaluation.
5:36:07And so with the law firm, for example, we have this really unique situation
5:36:10where we started this law firm from nothing.
5:36:14So we've kind of grown together.
5:36:15They've grown with Norm together.
5:36:16And so we work directly with lawyers to stand up these benchmarks and evaluations.
5:36:20and effectively, rather than think of a traditional roadmap,
5:36:24like when are you going to build an agent that can go do these three things,
5:36:27let's have a scorecard that determines when an agent,
5:36:30or when a human even, is successful at completing these tasks
5:36:34across various dimensions.
5:36:35And then we want to be able to see this just improve over time.
5:36:38So as we make general improvements to the system,
5:36:39we should see an uplift to these things like an attorney would expect
5:36:43to improve their workflow, but at the same time,
5:36:46not getting kind of going back to the state that we were originally,
5:36:49where we had added a lot of structure to it
5:36:51in a way that ultimately we found
5:36:53that we actually could get even better performance
5:36:55if we were to remove that upfront structure.
5:36:57I think another thing that's important to think through
5:36:59is you can produce, like I was saying,
5:37:02a lot more proof of concepts, proof of value very quickly.
5:37:06I still think the hard yards for a lot of development out there
5:37:09is getting it to production.
5:37:10And what I mean by that is infrastructure build-out
5:37:13that actually runs inside of the infrastructure
5:37:15that's available inside of any financial services
5:37:17or enterprise company, for that matter, out there.
5:37:20I think that is a challenge that a lot of folks underestimate.
5:37:23If you're selling AI tools, that is hard on the other side,
5:37:27just so you know.
5:37:27Deal with it quite a bit.
5:37:29The other thing is just dealing with the cybersecurity side of the fence
5:37:33and the supportability side of the fence.
5:37:35And so one thing I've noticed,
5:37:37the agents are getting better at this now,
5:37:39but it's making sure that the actual software that's being written
5:37:42is truly traceable, and it feeds into your SRE platform
5:37:46so that it can be monitorable in production.
5:37:48Those are things still that I like to have people think about.
5:37:51It's like they haven't started to scale yet with agents.
5:37:53They will, but it's something that, from my perspective,
5:37:56I think about regularly.
5:37:58Makes sense, and we are approached to this,
5:38:00and I'm sure a lot of other companies,
5:38:01and maybe even you guys have taken this approach as well.
5:38:04There's parts of our code base that we still require
5:38:08complete human oversight from when it comes to designing,
5:38:11building, code ownership,
5:38:13even approving any kind of changes to the code base,
5:38:15and these are our core primitives and these are things where basically you can build things
5:38:20on top of these but the things you build on top of these are not allowed to change these
5:38:23systems without having human review.
5:38:25It's because of the things you mentioned.
5:38:27Data security for us is as paramount as delivering quick, it's actually more important.
5:38:33For where you are and whether it's a consumer product or for us with obviously working with
5:38:37highly sensitive data, we have to build in so much infrastructure around this and ensure
5:38:42that we're not ever getting out ahead of these parts of the stack, which it can be hard because
5:38:47maybe somebody brings to you, and I'm sure you've seen this, a really cool demo and it's
5:38:50really exciting and you're looking at it terrified.
5:38:52So this is very cool, but to get to this point in terms of capabilities, it would expose
5:38:57so much that we're not able to do it.
5:38:58So I think it's just always that tradeoff.
5:39:00So we just basically invest heavily in these things and then invest conservatively in places
5:39:05where we realize as the frontier continues to improve, we want to make sure we have
5:39:10conduits. The way I described it
5:39:11actually this morning with the team is like basically
5:39:14throughout our platform we want to have like conduits
5:39:16for what's coming from the frontier. You don't want to have
5:39:18dams where like the intelligence is coming
5:39:20in and it can't quite get into your stack because you
5:39:22built things in such a way. Again, it's a great idea.
5:39:24We're going to get some of it wrong, some of it right. Just being very
5:39:26cognizant of it, talking about it a lot
5:39:27and making sure that our core pieces
5:39:30are locked down but then also
5:39:31being very open about just
5:39:33rip and replace. I think that's like a thing
5:39:35that I think everybody's gotten really good at.
5:39:37I think that's like one of the biggest
5:39:39I mean, I've been doing this for 27 years.
5:39:41Sometimes I feel like it's been like 100 years.
5:39:43Other times I feel like it's been five.
5:39:45I think that the thing that's like really made a material difference,
5:39:49like when I look at the ability to write software today
5:39:51versus what I was doing 10 years ago,
5:39:54was 10 years ago it took so much time, effort, and money
5:39:57in people's intellectual capital
5:40:00to produce a piece of software. It was almost like the ship had sailed. By the time you designed it and started writing code, it was out to sea and you were going to ride that baby for as long as you could until it got to the other side of the ocean.
5:40:10and like what I've become very comfortable with today is like for the most part a lot of like
5:40:16internal applications can be just thrown away it's build them have them work for a purpose and a point
5:40:22in time but understand that they're going to disappear at some point in time now much more
5:40:27quickly than ever before and then just think about like what comes next and so I think one of the
5:40:32challenges with like sitting in my seat or any like good engineer seat should be constantly
5:40:36questioning, should I rewrite this? And does it cost me less to rewrite it than it costs me to
5:40:42keep versioning the current version forward? And that is a question that a decade ago never would
5:40:47have came up in anyone's head if you went to say, I want to rewrite an internal application.
5:40:51Your CFO would have looked at you like you were insane. And I think now it's a completely relevant
5:40:55thing that all of us should be doing because net new applications might produce way more business
5:41:00value at the end of the day. So it's just something I encourage folks to do. It sounds like you guys
5:41:03are doing quite a bit at Norm.
5:41:05Yeah, exactly.
5:41:06And I think it's just that concept
5:41:07of what used to be expensive
5:41:08that's now cheap.
5:41:09And if you don't update quickly on that,
5:41:11then it's incredible.
5:41:13You said it feels like 27 or 5 years
5:41:15or 100 years or 5 years.
5:41:16It's like if you overindex too hard
5:41:19on something that was in June
5:41:20and now it's October,
5:41:21it feels like in some ways
5:41:23there's been 10 years
5:41:24of technological advancement.
5:41:26Well, great.
5:41:26We only have a minute or so left.
5:41:28Is there any closing remarks?
5:41:29I think the one thing
5:41:31I'd leave the audience on
5:41:32if you want good takeaways from this.
5:41:36We talked about a lot of very interesting things.
5:41:38What makes me excited to go to work
5:41:41is it's cool to get back to engineering.
5:41:43One of the things that I value is
5:41:46I was a hardcore low-level engineer for 10 years at IBM
5:41:49and wrote, this was way back in the day,
5:41:51of using Notepad and VI to write instruction-level code
5:41:55for RISC machines.
5:41:56It was awesome.
5:41:57It's nerdy as all heck.
5:41:58No one does this stuff anymore,
5:41:59but it was fun for me.
5:42:00I still enjoy those days.
5:42:01as you get more senior in your career and like the barrier to actually writing code on a daily basis
5:42:08disappears or it becomes more like hard to get through like dealing with a stream like processing
5:42:13stuff off a stream it's just all these things are like oh my god i can't even get started because
5:42:17the scaffolding is so hard with like ai it's been like reinvigorating to get back and actually build
5:42:23stuff because it's like i never lost my knowledge on how to architect things i see this with a lot
5:42:27of engineers out there, they understand because they've been in the market 10, 20, 30 years,
5:42:32they understand the complexity of system design, which hasn't changed.
5:42:36And now they can actually express that knowledge with these new cool agents.
5:42:39And so I think it's a super exciting time to be like starting a company, working at
5:42:42a company and like building software because there's like a whole new world that we can
5:42:47create out there.
5:42:47So that's what like excites me.
5:42:49Yeah.
5:42:49Well, thank you so much.
5:42:50It's been an honor.
5:42:51Yep.
5:42:51Thank you guys for having us here today.
5:42:52I appreciate it.
5:42:53Thank you.
