1
00:00:00,561 --> 00:00:05,861
I'm Scott Worland, I'm the CTO of Norm Ai and I'm privileged enough to be sharing the

2
00:00:05,861 --> 00:00:09,121
stage here with John Stecher who's the CTO of Blackstone.

3
00:00:09,861 --> 00:00:13,861
From what I understand, John has basically three full-time jobs, it sounds like.

4
00:00:14,101 --> 00:00:14,241
Yep.

5
00:00:14,601 --> 00:00:19,361
Responsible for all of the technology at Blackstone, also advises on their investments,

6
00:00:20,521 --> 00:00:24,901
and serves as a resource for the portfolio companies for all things technology.

7
00:00:25,141 --> 00:00:25,861
Did I miss anything?

8
00:00:25,941 --> 00:00:27,161
Nope, I hit it all perfectly, Matt.

9
00:00:27,801 --> 00:00:28,501
Cool job.

10
00:00:29,021 --> 00:00:31,241
Yeah, and if you're not familiar with what Blackstone does,

11
00:00:31,461 --> 00:00:33,481
then you probably have wandered into the wrong conference.

12
00:00:34,181 --> 00:00:36,641
So I'm not going to go into that, but you may not know what Norm does.

13
00:00:36,961 --> 00:00:41,621
So really briefly, Norm is an AI company, and we operate in the legal and regulatory space.

14
00:00:41,841 --> 00:00:47,581
We have an enterprise AI platform that's deployed in some of the largest financial institutions in the world,

15
00:00:47,581 --> 00:00:50,461
and then we also have an affiliated law firm, Norm Law,

16
00:00:50,721 --> 00:00:52,881
and our AI platform also supports that.

17
00:00:54,961 --> 00:00:56,761
So since the end of last year,

18
00:00:56,961 --> 00:00:58,441
and certainly all of this year,

19
00:00:58,841 --> 00:01:01,621
everyone's been racing to figure out how to deploy AI agents.

20
00:01:01,881 --> 00:01:04,181
And I'd love to hear how you're thinking about this at Blackstone

21
00:01:04,181 --> 00:01:05,721
and how you're trying to make sure you get this right.

22
00:01:05,901 --> 00:01:08,261
Yeah, I mean, I think the race has definitely been on

23
00:01:08,261 --> 00:01:10,361
as the technology has improved mightily

24
00:01:10,361 --> 00:01:12,921
since really like February, March of last year,

25
00:01:13,081 --> 00:01:14,921
just in terms of context windows,

26
00:01:14,921 --> 00:01:19,101
being able to run stuff for significantly longer amounts of time have like resulted in people being

27
00:01:19,101 --> 00:01:23,781
able to produce just more value and one of the things that like we've been focused on a lot and

28
00:01:23,781 --> 00:01:28,161
i think every company in our portfolio i advise on and work with all the time that they need to

29
00:01:28,161 --> 00:01:33,301
have top of mind for like being successful with ai agents in general is you have to get your data

30
00:01:33,301 --> 00:01:38,121
story right and so like one of the biggest things impediments and tripping points that i see for

31
00:01:38,121 --> 00:01:42,741
companies out there when they want they're all in on ai they want to build as many agents as possible

32
00:01:42,741 --> 00:01:44,621
They really want to kind of extract value.

33
00:01:44,681 --> 00:01:49,741
If they don't have a cohesive data strategy that's moving stuff out of transactional systems

34
00:01:49,741 --> 00:01:54,761
into something that can be accessed and easily managed and recalled from like an agent perspective,

35
00:01:54,761 --> 00:01:56,421
it is going to fail.

36
00:01:56,561 --> 00:01:59,081
And so that's been one of the biggest things that we've been focused on.

37
00:01:59,221 --> 00:02:03,981
You know, as a big asset manager, we have a ton of data, proprietary data, public data,

38
00:02:04,121 --> 00:02:04,801
all woven together.

39
00:02:04,921 --> 00:02:10,041
We have to move that out of systems into something that we can really leverage from an AI perspective.

40
00:02:10,041 --> 00:02:19,561
I think the second thing that has nothing to do with AI, but you can leverage AI to actually figure out how to do the best possible way, is straight through processing still matters.

41
00:02:19,761 --> 00:02:24,801
I'm always shocked at how everybody just talks about agents as being the biggest differentiator in the world.

42
00:02:25,101 --> 00:02:37,381
The thing that I've leveraged myself using Factory and a couple other ways to go off and build code inside of the firm is taking time, having agents help to craft something for me, understand the workflows out there that people are doing,

43
00:02:37,381 --> 00:02:41,741
and then make sure that I'm actually writing code to optimize for straight-through processing

44
00:02:41,741 --> 00:02:45,181
and then using agents to inject in intelligence where necessary,

45
00:02:45,181 --> 00:02:49,041
where humans were making decisions in the past for a variety of rote work things.

46
00:02:49,161 --> 00:02:53,321
So that's really the two key elements that I spend a ton of time thinking about every single day.

47
00:02:53,441 --> 00:02:57,121
And then the biggest thing from an agent perspective is really stepping back

48
00:02:57,121 --> 00:03:02,781
and looking at what is work that we want to build agents to do better,

49
00:03:02,781 --> 00:03:05,741
to offload intellectual cycles

50
00:03:05,741 --> 00:03:07,221
from our investment professionals,

51
00:03:07,401 --> 00:03:08,781
our finance professionals, our engineers,

52
00:03:09,061 --> 00:03:10,981
such that they can do higher value work.

53
00:03:11,061 --> 00:03:11,841
And there's a ton of stuff

54
00:03:11,841 --> 00:03:13,621
when you think about the software development life cycle,

55
00:03:14,061 --> 00:03:15,961
writing test cases, functional test cases,

56
00:03:16,121 --> 00:03:17,061
performance test cases,

57
00:03:18,101 --> 00:03:21,501
versioning forward libraries underneath applications.

58
00:03:21,721 --> 00:03:23,641
It's the bane of every engineer's existence.

59
00:03:23,861 --> 00:03:25,821
I've hated it for 27 years that I've been doing this.

60
00:03:26,201 --> 00:03:27,401
Having an agent be able to go off

61
00:03:27,401 --> 00:03:28,261
and do these things for me

62
00:03:28,261 --> 00:03:29,941
and renovate my code and my source code

63
00:03:29,941 --> 00:03:31,401
has made all the difference in the world

64
00:03:31,401 --> 00:03:32,721
for the velocity that we deliver at.

65
00:03:33,061 --> 00:03:34,501
So how are you thinking about it at Norm?

66
00:03:35,061 --> 00:03:38,161
Yeah, so Norm was founded in 2023.

67
00:03:38,441 --> 00:03:40,241
So, you know, that's three years,

68
00:03:40,501 --> 00:03:42,301
I guess four years after GPT-1,

69
00:03:42,501 --> 00:03:44,401
but still a year before there was like,

70
00:03:44,521 --> 00:03:45,301
you know, chain of thought.

71
00:03:45,441 --> 00:03:46,501
So it was like an interesting time

72
00:03:46,501 --> 00:03:47,321
in terms of development.

73
00:03:47,581 --> 00:03:49,781
I remember when we had our first team get together,

74
00:03:49,961 --> 00:03:51,681
we were at Stanford's campus for a week.

75
00:03:51,821 --> 00:03:53,501
And I mean, I'm probably misremembering

76
00:03:53,501 --> 00:03:54,161
the exact timeline,

77
00:03:54,261 --> 00:03:57,081
but it felt like we went from 4K to 8K

78
00:03:57,081 --> 00:03:59,441
to 128K context windows.

79
00:03:59,621 --> 00:04:00,821
And then we had multimodal

80
00:04:00,821 --> 00:04:02,741
and everything felt like it was coming all at once.

81
00:04:03,321 --> 00:04:06,961
And then there was a bit of a time where things did slow down when it was coming from the

82
00:04:06,961 --> 00:04:07,301
frontier.

83
00:04:07,681 --> 00:04:12,941
I think like a lot of companies in early 2024, we were trying to deploy these models as part

84
00:04:12,941 --> 00:04:15,241
of our workflow and as part of our production software.

85
00:04:15,841 --> 00:04:21,761
We ended up adding a lot of structure to the system and this comes from all different parts

86
00:04:21,761 --> 00:04:22,241
of the stack.

87
00:04:22,741 --> 00:04:27,601
Whether that was from how we went and took the content that we knew needed to be eventually

88
00:04:27,601 --> 00:04:32,201
added to context to the models and we had very specific ways in which we did that. Or

89
00:04:32,201 --> 00:04:37,021
if there was the sequencing of the models, if there was steps we needed to do in terms

90
00:04:37,021 --> 00:04:41,141
of looking for consistency and then ultimately distilling everything down into some useful

91
00:04:41,141 --> 00:04:45,161
output, we ended up with something that looked like a large map reduce problem effectively.

92
00:04:45,881 --> 00:04:49,781
We had a lot of scaffolding for the agents and we were very particular about how these

93
00:04:51,181 --> 00:04:56,501
models ran. It was much more like a workflow. What this allowed us to do was get really

94
00:04:57,041 --> 00:05:02,281
high quality and also very accurate, but also just very consistent, which is very important

95
00:05:02,281 --> 00:05:04,361
for us in the regulatory space, outputs.

96
00:05:05,581 --> 00:05:09,541
But then one fortuitous thing as part of doing this, because we wanted to have some way to

97
00:05:09,541 --> 00:05:13,861
regression test this big system, we ended up with a really large set of evaluation samples.

98
00:05:14,041 --> 00:05:17,821
We had tens of thousands, actually, of hand-labeled examples by experts.

99
00:05:18,001 --> 00:05:22,181
And so as soon as we moved to a place where it was clear that agents were actually a thing,

100
00:05:22,181 --> 00:05:28,101
and by agents I simply mean, you know, given like an LLM, a system, or given a set of tools

101
00:05:28,101 --> 00:05:32,041
that it can use, give it some goal and give it some way to verify itself against that

102
00:05:32,041 --> 00:05:35,241
goal, allowing it just to go figure out how to do the thing in the best way possible.

103
00:05:35,821 --> 00:05:40,661
We were able to just take this benchmark set and just immediately run on top of it with

104
00:05:40,661 --> 00:05:41,801
some of these more agentic approaches.

105
00:05:41,981 --> 00:05:45,841
And we saw, you know, I think a lot of people had this moment where you're staring at hundreds

106
00:05:45,841 --> 00:05:48,981
of thousands of lines of code your team had really thoughtfully written and realizing

107
00:05:48,981 --> 00:05:51,581
most of it was pretty obsolete and was just kind of actually in the way.

108
00:05:51,581 --> 00:05:57,321
And so to update quickly on this, what we did was we took, I got the engineering leadership team.

109
00:05:57,501 --> 00:06:01,061
We actually rented a Sprinter van and we drove to upstate New York.

110
00:06:01,161 --> 00:06:02,561
It was actually after a big snowstorm.

111
00:06:02,701 --> 00:06:05,581
We had an Airbnb and it was like, forget norm for a minute.

112
00:06:05,841 --> 00:06:12,081
Let's just say what would we build today as far as a platform, knowing what we know right now, if we were going to be starting over.

113
00:06:12,301 --> 00:06:18,221
And then from that, we left with basically a set of specs about like what this platform needed to be able to do to enable two things.

114
00:06:18,221 --> 00:06:21,261
one, deliver agentic experiences to our clients,

115
00:06:21,421 --> 00:06:23,941
and then also it needed to be amenable

116
00:06:23,941 --> 00:06:26,061
to having AI as co-developers

117
00:06:26,061 --> 00:06:27,621
alongside of our own development team.

118
00:06:27,841 --> 00:06:30,461
And that's what we've been executing against ever since then.

119
00:06:30,641 --> 00:06:32,121
So I think this...

120
00:06:32,121 --> 00:06:32,661
Do you have a question?

121
00:06:32,761 --> 00:06:33,401
No, I was going to say,

122
00:06:33,461 --> 00:06:34,981
I think you hit on a key point, Scott,

123
00:06:35,101 --> 00:06:37,081
where it says the eval side of the fence

124
00:06:37,081 --> 00:06:38,121
that you were talking about before,

125
00:06:38,281 --> 00:06:39,781
if you're working in financial services,

126
00:06:40,981 --> 00:06:44,221
the quality bar for what agents have to deliver on

127
00:06:44,221 --> 00:06:45,141
is very, very high.

128
00:06:45,321 --> 00:06:47,361
And so I think really thinking through

129
00:06:47,361 --> 00:06:53,881
like what is your eval set for how you basically test your agents, whatever work they're performing,

130
00:06:54,081 --> 00:06:58,681
that the output makes sense, and it's consistent and repeatable. And I've seen across the board,

131
00:06:58,821 --> 00:07:03,881
as model versions change underneath the covers, you get different answers. You get skew in terms

132
00:07:03,881 --> 00:07:08,041
of the accuracy of information that comes back. So having good evals so that you can run those

133
00:07:08,041 --> 00:07:12,881
consistently and really measure the health of a system is paramount to success. Same thing happens

134
00:07:12,881 --> 00:07:17,001
if harnesses change that you're running stuff on top of. You'll get different answers. Data change

135
00:07:17,001 --> 00:07:21,341
different answers. So it's something where I think if you're trafficking in financial services,

136
00:07:21,961 --> 00:07:27,541
like the duty to be accurate is like up here and hitting that is like something like we think about

137
00:07:27,541 --> 00:07:31,421
all the time. Definitely. I think this leads nicely into this concept that we've been talking

138
00:07:31,421 --> 00:07:34,981
about just like with the bitter lesson. So like everyone's familiar with it

139
00:07:35,481 --> 00:07:46,481
There's this idea that these handcrafted systems built by clever people thinking hard about the problem will oftentimes eventually be dominated by systems that scale with compute.

140
00:07:47,821 --> 00:07:52,421
I think Rich Sutton originally talked about this in 2019, but this has been since the beginning of neural networks.

141
00:07:52,561 --> 00:07:58,821
If you go back to the 50s and 60s, like with Rosenblatt's little perceptron, I think the New York Times ran an article or somebody did saying

142
00:07:58,821 --> 00:08:01,781
It was the first machine that could ever have its own original thoughts.

143
00:08:02,481 --> 00:08:09,781
And there's actually quite a fascinating history here of how the expert systems and the connectionists and what ended up happening.

144
00:08:09,861 --> 00:08:14,101
A lot of it actually was highly litigated in England, and a lot of AI research was dismantled.

145
00:08:14,541 --> 00:08:22,181
But one funny thing, if you go back to the 80s even, there was a lead at IBM's translation lab, and he made a comment, which is very much the bitter lesson.

146
00:08:22,461 --> 00:08:24,401
Every time I fire a linguist, my models improve.

147
00:08:24,401 --> 00:08:27,901
The point of having these linguists in there and trying to think about the rules of language

148
00:08:27,901 --> 00:08:30,481
versus training a statistical model based on probabilities.

149
00:08:30,941 --> 00:08:33,401
We speed up to today and we see this all the time.

150
00:08:33,541 --> 00:08:35,201
We've been touching on lots of points about this.

151
00:08:35,281 --> 00:08:38,901
I'm curious, when you think about your roadmaps, when people on your team are bringing you

152
00:08:38,901 --> 00:08:43,161
project ideas, how do you keep this in mind as you're thinking about this for Blackstone?

153
00:08:43,361 --> 00:08:48,221
A lot of what we do is like the simple equation is giving enough compute.

154
00:08:48,701 --> 00:08:53,841
Every problem is solvable by hardware and software at some point in time in the future.

155
00:08:53,841 --> 00:08:55,761
It's Elon talks about this energy wise, et cetera.

156
00:08:55,921 --> 00:08:57,341
So it's like, it's a big map.

157
00:08:57,541 --> 00:08:58,401
It's a whole bunch of stuff.

158
00:08:58,541 --> 00:08:59,621
How do you bring it back to reality?

159
00:09:00,041 --> 00:09:04,561
Like what I spend a lot of my time doing from like my seat, working with engineers and my

160
00:09:04,561 --> 00:09:09,241
team with product people, with our SRE function, with like stakeholders is really trying to

161
00:09:09,241 --> 00:09:13,621
define like, you know, what is the ROI of what I'm being asked to work on?

162
00:09:13,801 --> 00:09:17,441
And so like when I think of building agents, I think about, you know, getting data into

163
00:09:17,441 --> 00:09:20,641
the right spot, think about building dashboards, trading platforms, whatever it is.

164
00:09:20,641 --> 00:09:26,401
it's like if I look at like what's the ROI that's going to be created from the actual ask that's

165
00:09:26,401 --> 00:09:31,341
coming in to me from a team perspective and then like what's the what's the true cost of actually

166
00:09:31,341 --> 00:09:36,261
building something today like how can I balance that equation and then I make a decision of like

167
00:09:36,261 --> 00:09:40,661
you know is this something where the models are going to get good enough I'm going to have more

168
00:09:40,661 --> 00:09:44,341
than enough compute in six months that it's not worth trying to tackle the problem today

169
00:09:44,341 --> 00:09:49,341
it's better to delay it and actually address it with one of the new models we have made that

170
00:09:49,341 --> 00:09:53,761
decision before is like let's actually wait for the model to become slightly more sophisticated

171
00:09:53,761 --> 00:09:58,621
so that we can solve these problems with it because it will cost less at that time and the

172
00:09:58,621 --> 00:10:04,101
need for that application to be built today isn't pressing there's no like real business value

173
00:10:04,101 --> 00:10:08,981
increase that can be like will be created because it's produced today versus six months from now so

174
00:10:08,981 --> 00:10:14,301
that's a trade-off that we actually you know make regularly and i think about regularly um the other

175
00:10:14,301 --> 00:10:18,261
thing that we do is like you don't want to wait too long to basically build something and so you

176
00:10:18,261 --> 00:10:21,501
can always be waiting for the new model, waiting for the new technology or whatever else.

177
00:10:21,861 --> 00:10:26,301
The beautiful thing with all of the various different software development harnesses today

178
00:10:26,301 --> 00:10:32,141
is if you lean into them, it's amazing how quick you can go from concept, mapping out your planning

179
00:10:32,141 --> 00:10:37,661
and concept with Fable for just, hey, work with me and let's define this with the product person

180
00:10:37,661 --> 00:10:42,501
or with the stakeholder, to actually writing code, to having a workable proof of concept.

181
00:10:42,661 --> 00:10:45,741
And when I say workable proof of concept, it's something people could click through, interact

182
00:10:45,741 --> 00:10:49,721
with whatever else it is, doesn't have cybersecurity review, doesn't have all the other things that

183
00:10:49,721 --> 00:10:53,101
you need to do in our industry to make sure you put something into production that scales,

184
00:10:53,161 --> 00:10:59,141
but you can get to a POC or a POV very quickly. And so that's one big thing I try to encourage

185
00:10:59,141 --> 00:11:04,041
all my engineers to do frequently is just get to POC. Don't worry about PowerPoints. Don't

186
00:11:04,041 --> 00:11:08,561
worry about anything else. Leverage tokens, which I think it's a fallacy to be like counting tokens

187
00:11:08,561 --> 00:11:12,741
in some way, shape, or form. It's better to think about producing value. Go off produce value,

188
00:11:12,741 --> 00:11:17,821
produce POVs, put it in front of people, iterate on it quickly, and throw it away if you don't need

189
00:11:17,821 --> 00:11:21,721
it. You will spend far less time and money doing those types of things. And so that's like a big

190
00:11:21,721 --> 00:11:26,381
way that we think about trying to solve it. It doesn't maps into the paradigm, like the, you

191
00:11:26,381 --> 00:11:30,521
know, how do you deal with increasing compute, making it easier to do these things. But like,

192
00:11:30,561 --> 00:11:34,761
I just, I think in a much shorter timeframe. Yeah. It's so interesting. Cause it's like,

193
00:11:35,521 --> 00:11:39,361
you know, I mean, you touched on so many things that I think everybody's probably been thinking

194
00:11:39,361 --> 00:11:43,481
about too. And if you go back to 2023, when we were originally like sitting around as a small

195
00:11:43,481 --> 00:11:46,981
group talking about what we're building, we were aware of the bitter lesson then. We talked about

196
00:11:46,981 --> 00:11:50,621
it. We said, how do we not get in this trap? And I think at this point, it's just really evident to

197
00:11:50,621 --> 00:11:54,241
me that it's like, it's definitely a gradient, right? It's not this binary thing about you can

198
00:11:54,241 --> 00:11:58,641
somehow build the perfect technology to index into all the, you know, everything coming in from the

199
00:11:58,641 --> 00:12:02,801
frontier in a way that you don't ever have to throw things away. So really it gets down to like,

200
00:12:02,921 --> 00:12:07,361
where do we want to invest heavily and where are we willing to invest conservatively? Because like

201
00:12:07,361 --> 00:12:11,681
you said, we need something today. But at the same time, we're just aware that some of these

202
00:12:11,681 --> 00:12:16,261
investments we make, particularly around things like we've built fairly complicated systems around

203
00:12:16,261 --> 00:12:21,661
all of our agent primitives. So the harness, the sandbox, how egress works, all of these things.

204
00:12:21,801 --> 00:12:26,441
And we also want to make sure that as things are coming out either from a frontier lab or from a

205
00:12:26,441 --> 00:12:31,121
third party, that we're able to index into that. And so I think a lot of it is just kind of when

206
00:12:31,121 --> 00:12:35,541
do you update? And that's one of the hardest things to do. And for me, a good example was

207
00:12:35,541 --> 00:12:38,101
I updated really hard on Claude Code.

208
00:12:38,421 --> 00:12:41,621
Not so much that it was suddenly tools in my profession

209
00:12:41,621 --> 00:12:43,161
that we could use that were really incredible.

210
00:12:43,581 --> 00:12:45,561
What it really was for me was this idea,

211
00:12:45,601 --> 00:12:47,281
because I was a bit skeptical up to that point

212
00:12:47,281 --> 00:12:49,421
of how we would get something that was truly generalizable.

213
00:12:49,921 --> 00:12:51,721
It's like, well, what does Claude Code give you?

214
00:12:51,841 --> 00:12:54,021
Well, it can write code, and then it can execute the code,

215
00:12:54,121 --> 00:12:56,741
and basically run a computer in some verifiable way.

216
00:12:57,121 --> 00:12:59,441
At that moment, I was like, well, we have all the ingredients

217
00:12:59,441 --> 00:13:01,421
for a truly generalizable intelligence,

218
00:13:01,541 --> 00:13:02,961
because if you can go do this at scale,

219
00:13:02,961 --> 00:13:09,521
then, I mean, you know, of course, like you can basically do anything in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today.

220
00:13:09,641 --> 00:13:14,201
And you fast forward to now and like all these tools that people are using is like AI assistants and things.

221
00:13:14,281 --> 00:13:18,541
This is why they're so amazing is because you can ask a question it's never seen before, fully out of distribution.

222
00:13:18,961 --> 00:13:21,061
It can spin up a computer and go do this thing in the background.

223
00:13:21,081 --> 00:13:25,101
And for a consumer, it's amazing. And we're also seeing this now within the enterprise, too.

224
00:13:25,201 --> 00:13:28,701
And so for me, that moment was like, well, where are we going to invest heavily?

225
00:13:28,701 --> 00:13:31,341
Of course, we still need to invest in what we're building on the agent side.

226
00:13:31,881 --> 00:13:36,441
But going back to what John mentioned, how we do our data capture and distillation,

227
00:13:36,641 --> 00:13:38,821
and how we make this available as state to these models,

228
00:13:38,901 --> 00:13:41,941
and then also just how we do our benchmarks and evaluation.

229
00:13:42,301 --> 00:13:45,621
And so with the law firm, for example, we have this really unique situation

230
00:13:45,621 --> 00:13:48,821
where we started this law firm from nothing.

231
00:13:49,061 --> 00:13:50,361
So we've kind of grown together.

232
00:13:50,481 --> 00:13:51,401
They've grown with Norm together.

233
00:13:51,621 --> 00:13:55,861
And so we work directly with lawyers to stand up these benchmarks and evaluations.

234
00:13:55,861 --> 00:13:58,981
and effectively, rather than think of a traditional roadmap,

235
00:13:59,261 --> 00:14:02,001
like when are you going to build an agent that can go do these three things,

236
00:14:02,401 --> 00:14:04,821
let's have a scorecard that determines when an agent,

237
00:14:05,021 --> 00:14:09,221
or when a human even, is successful at completing these tasks

238
00:14:09,221 --> 00:14:10,241
across various dimensions.

239
00:14:10,701 --> 00:14:12,861
And then we want to be able to see this just improve over time.

240
00:14:13,061 --> 00:14:14,801
So as we make general improvements to the system,

241
00:14:14,941 --> 00:14:18,881
we should see an uplift to these things like an attorney would expect

242
00:14:18,881 --> 00:14:21,341
to improve their workflow, but at the same time,

243
00:14:21,481 --> 00:14:24,401
not getting kind of going back to the state that we were originally,

244
00:14:24,401 --> 00:14:26,701
where we had added a lot of structure to it

245
00:14:26,701 --> 00:14:28,421
in a way that ultimately we found

246
00:14:28,421 --> 00:14:30,041
that we actually could get even better performance

247
00:14:30,041 --> 00:14:31,921
if we were to remove that upfront structure.

248
00:14:32,321 --> 00:14:34,861
I think another thing that's important to think through

249
00:14:34,861 --> 00:14:37,021
is you can produce, like I was saying,

250
00:14:37,021 --> 00:14:41,021
a lot more proof of concepts, proof of value very quickly.

251
00:14:41,141 --> 00:14:44,321
I still think the hard yards for a lot of development out there

252
00:14:44,321 --> 00:14:45,781
is getting it to production.

253
00:14:45,961 --> 00:14:48,181
And what I mean by that is infrastructure build-out

254
00:14:48,181 --> 00:14:50,241
that actually runs inside of the infrastructure

255
00:14:50,241 --> 00:14:52,821
that's available inside of any financial services

256
00:14:52,821 --> 00:14:55,441
or enterprise company, for that matter, out there.

257
00:14:55,561 --> 00:14:58,221
I think that is a challenge that a lot of folks underestimate.

258
00:14:58,441 --> 00:15:02,261
If you're selling AI tools, that is hard on the other side,

259
00:15:02,361 --> 00:15:02,841
just so you know.

260
00:15:03,001 --> 00:15:04,021
Deal with it quite a bit.

261
00:15:04,441 --> 00:15:08,601
The other thing is just dealing with the cybersecurity side of the fence

262
00:15:08,601 --> 00:15:10,141
and the supportability side of the fence.

263
00:15:10,601 --> 00:15:12,141
And so one thing I've noticed,

264
00:15:12,821 --> 00:15:14,841
the agents are getting better at this now,

265
00:15:14,941 --> 00:15:17,621
but it's making sure that the actual software that's being written

266
00:15:17,621 --> 00:15:21,081
is truly traceable, and it feeds into your SRE platform

267
00:15:21,081 --> 00:15:23,121
so that it can be monitorable in production.

268
00:15:23,281 --> 00:15:25,881
Those are things still that I like to have people think about.

269
00:15:26,061 --> 00:15:28,481
It's like they haven't started to scale yet with agents.

270
00:15:28,481 --> 00:15:31,201
They will, but it's something that, from my perspective,

271
00:15:31,241 --> 00:15:32,681
I think about regularly.

272
00:15:33,281 --> 00:15:35,081
Makes sense, and we are approached to this,

273
00:15:35,141 --> 00:15:36,461
and I'm sure a lot of other companies,

274
00:15:36,581 --> 00:15:38,541
and maybe even you guys have taken this approach as well.

275
00:15:39,021 --> 00:15:42,941
There's parts of our code base that we still require

276
00:15:43,681 --> 00:15:46,641
complete human oversight from when it comes to designing,

277
00:15:46,921 --> 00:15:48,341
building, code ownership,

278
00:15:48,701 --> 00:15:50,741
even approving any kind of changes to the code base,

279
00:15:50,741 --> 00:15:55,141
and these are our core primitives and these are things where basically you can build things

280
00:15:55,141 --> 00:15:58,541
on top of these but the things you build on top of these are not allowed to change these

281
00:15:58,541 --> 00:15:59,841
systems without having human review.

282
00:16:00,321 --> 00:16:01,621
It's because of the things you mentioned.

283
00:16:02,341 --> 00:16:07,941
Data security for us is as paramount as delivering quick, it's actually more important.

284
00:16:08,301 --> 00:16:12,921
For where you are and whether it's a consumer product or for us with obviously working with

285
00:16:12,921 --> 00:16:17,501
highly sensitive data, we have to build in so much infrastructure around this and ensure

286
00:16:17,501 --> 00:16:22,201
that we're not ever getting out ahead of these parts of the stack, which it can be hard because

287
00:16:22,201 --> 00:16:25,261
maybe somebody brings to you, and I'm sure you've seen this, a really cool demo and it's

288
00:16:25,261 --> 00:16:27,221
really exciting and you're looking at it terrified.

289
00:16:27,961 --> 00:16:32,441
So this is very cool, but to get to this point in terms of capabilities, it would expose

290
00:16:32,441 --> 00:16:33,841
so much that we're not able to do it.

291
00:16:33,921 --> 00:16:35,581
So I think it's just always that tradeoff.

292
00:16:35,721 --> 00:16:40,641
So we just basically invest heavily in these things and then invest conservatively in places

293
00:16:40,641 --> 00:16:45,101
where we realize as the frontier continues to improve, we want to make sure we have

294
00:16:45,661 --> 00:16:46,861
conduits. The way I described it

295
00:16:46,861 --> 00:16:49,121
actually this morning with the team is like basically

296
00:16:49,121 --> 00:16:51,481
throughout our platform we want to have like conduits

297
00:16:51,481 --> 00:16:53,161
for what's coming from the frontier. You don't want to have

298
00:16:53,161 --> 00:16:55,181
dams where like the intelligence is coming

299
00:16:55,181 --> 00:16:57,181
in and it can't quite get into your stack because you

300
00:16:57,181 --> 00:16:59,121
built things in such a way. Again, it's a great idea.

301
00:16:59,241 --> 00:17:01,181
We're going to get some of it wrong, some of it right. Just being very

302
00:17:01,181 --> 00:17:02,841
cognizant of it, talking about it a lot

303
00:17:02,841 --> 00:17:05,061
and making sure that our core pieces

304
00:17:05,061 --> 00:17:06,961
are locked down but then also

305
00:17:06,961 --> 00:17:08,941
being very open about just

306
00:17:08,941 --> 00:17:11,001
rip and replace. I think that's like a thing

307
00:17:11,001 --> 00:17:12,561
that I think everybody's gotten really good at.

308
00:17:12,741 --> 00:17:14,441
I think that's like one of the biggest

309
00:17:14,441 --> 00:17:16,861
I mean, I've been doing this for 27 years.

310
00:17:17,001 --> 00:17:18,621
Sometimes I feel like it's been like 100 years.

311
00:17:18,781 --> 00:17:20,061
Other times I feel like it's been five.

312
00:17:20,301 --> 00:17:24,181
I think that the thing that's like really made a material difference,

313
00:17:24,201 --> 00:17:26,921
like when I look at the ability to write software today

314
00:17:26,921 --> 00:17:28,761
versus what I was doing 10 years ago,

315
00:17:29,081 --> 00:17:32,961
was 10 years ago it took so much time, effort, and money

316
00:17:32,961 --> 00:17:35,021
in people's intellectual capital

317
00:17:35,101 --> 00:17:46,001
to produce a piece of software. It was almost like the ship had sailed. By the time you designed it and started writing code, it was out to sea and you were going to ride that baby for as long as you could until it got to the other side of the ocean.

318
00:17:46,001 --> 00:17:51,381
and like what I've become very comfortable with today is like for the most part a lot of like

319
00:17:51,381 --> 00:17:57,401
internal applications can be just thrown away it's build them have them work for a purpose and a point

320
00:17:57,401 --> 00:18:02,981
in time but understand that they're going to disappear at some point in time now much more

321
00:18:02,981 --> 00:18:07,761
quickly than ever before and then just think about like what comes next and so I think one of the

322
00:18:07,761 --> 00:18:11,661
challenges with like sitting in my seat or any like good engineer seat should be constantly

323
00:18:11,661 --> 00:18:17,441
questioning, should I rewrite this? And does it cost me less to rewrite it than it costs me to

324
00:18:17,441 --> 00:18:22,221
keep versioning the current version forward? And that is a question that a decade ago never would

325
00:18:22,221 --> 00:18:25,881
have came up in anyone's head if you went to say, I want to rewrite an internal application.

326
00:18:26,341 --> 00:18:30,401
Your CFO would have looked at you like you were insane. And I think now it's a completely relevant

327
00:18:30,401 --> 00:18:35,021
thing that all of us should be doing because net new applications might produce way more business

328
00:18:35,021 --> 00:18:38,661
value at the end of the day. So it's just something I encourage folks to do. It sounds like you guys

329
00:18:38,661 --> 00:18:39,861
are doing quite a bit at Norm.

330
00:18:40,601 --> 00:18:40,961
Yeah, exactly.

331
00:18:41,121 --> 00:18:42,441
And I think it's just that concept

332
00:18:42,441 --> 00:18:43,801
of what used to be expensive

333
00:18:43,801 --> 00:18:44,501
that's now cheap.

334
00:18:44,681 --> 00:18:46,361
And if you don't update quickly on that,

335
00:18:46,661 --> 00:18:48,541
then it's incredible.

336
00:18:48,561 --> 00:18:50,841
You said it feels like 27 or 5 years

337
00:18:50,841 --> 00:18:51,761
or 100 years or 5 years.

338
00:18:52,001 --> 00:18:54,581
It's like if you overindex too hard

339
00:18:54,581 --> 00:18:55,681
on something that was in June

340
00:18:55,681 --> 00:18:56,721
and now it's October,

341
00:18:57,001 --> 00:18:58,461
it feels like in some ways

342
00:18:58,461 --> 00:18:59,221
there's been 10 years

343
00:18:59,221 --> 00:19:00,161
of technological advancement.

344
00:19:01,381 --> 00:19:01,861
Well, great.

345
00:19:01,941 --> 00:19:03,401
We only have a minute or so left.

346
00:19:03,461 --> 00:19:04,481
Is there any closing remarks?

347
00:19:04,741 --> 00:19:06,061
I think the one thing

348
00:19:06,061 --> 00:19:07,941
I'd leave the audience on

349
00:19:07,941 --> 00:19:10,781
if you want good takeaways from this.

350
00:19:11,201 --> 00:19:13,361
We talked about a lot of very interesting things.

351
00:19:13,941 --> 00:19:16,301
What makes me excited to go to work

352
00:19:16,301 --> 00:19:18,561
is it's cool to get back to engineering.

353
00:19:18,801 --> 00:19:20,381
One of the things that I value is

354
00:19:21,201 --> 00:19:24,481
I was a hardcore low-level engineer for 10 years at IBM

355
00:19:24,481 --> 00:19:26,401
and wrote, this was way back in the day,

356
00:19:26,521 --> 00:19:30,201
of using Notepad and VI to write instruction-level code

357
00:19:30,201 --> 00:19:31,181
for RISC machines.

358
00:19:31,241 --> 00:19:32,001
It was awesome.

359
00:19:32,381 --> 00:19:33,321
It's nerdy as all heck.

360
00:19:33,441 --> 00:19:34,481
No one does this stuff anymore,

361
00:19:34,541 --> 00:19:35,501
but it was fun for me.

362
00:19:35,621 --> 00:19:36,641
I still enjoy those days.

363
00:19:36,641 --> 00:19:43,101
as you get more senior in your career and like the barrier to actually writing code on a daily basis

364
00:19:43,101 --> 00:19:48,901
disappears or it becomes more like hard to get through like dealing with a stream like processing

365
00:19:48,901 --> 00:19:52,061
stuff off a stream it's just all these things are like oh my god i can't even get started because

366
00:19:52,061 --> 00:19:58,081
the scaffolding is so hard with like ai it's been like reinvigorating to get back and actually build

367
00:19:58,081 --> 00:20:02,441
stuff because it's like i never lost my knowledge on how to architect things i see this with a lot

368
00:20:02,441 --> 00:20:07,301
of engineers out there, they understand because they've been in the market 10, 20, 30 years,

369
00:20:07,481 --> 00:20:10,661
they understand the complexity of system design, which hasn't changed.

370
00:20:11,061 --> 00:20:14,161
And now they can actually express that knowledge with these new cool agents.

371
00:20:14,301 --> 00:20:17,961
And so I think it's a super exciting time to be like starting a company, working at

372
00:20:17,961 --> 00:20:22,301
a company and like building software because there's like a whole new world that we can

373
00:20:22,301 --> 00:20:22,781
create out there.

374
00:20:22,881 --> 00:20:23,921
So that's what like excites me.

375
00:20:24,221 --> 00:20:24,381
Yeah.

376
00:20:24,601 --> 00:20:25,641
Well, thank you so much.

377
00:20:25,841 --> 00:20:26,221
It's been an honor.

378
00:20:26,361 --> 00:20:26,461
Yep.

379
00:20:26,581 --> 00:20:27,581
Thank you guys for having us here today.

380
00:20:27,681 --> 00:20:28,141
I appreciate it.

381
00:20:28,141 --> 00:20:30,521
Thank you.