# Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI

John Stecher, Blackstone · Scott Worland, Norm AI

[Talk page](https://ai.engineer/talks/ra8X7_bnujw-why-waiting-next-ai-model-pay-off) · [Video](https://www.youtube.com/watch?v=ra8X7_bnujw)

## What to Build Before the Next Model Arrives

Blackstone CTO John Stecher weighs the value of shipping six months early, while Norm AI CTO Scott Worland explains how tens of thousands of expert-labeled examples gave his team the confidence to replace software that worked.

### Two views of the same deployment decision

[0:12](https://www.youtube.com/watch?v=ra8X7_bnujw&t=12s)

John Stecher sees technology proposals from three positions at Blackstone. He runs the firm’s technology, advises on investments and serves as a technical resource for its portfolio companies. When a team proposes an agent, he has to consider what it could do for the people using it and what it would take for the organization to operate it. Scott Worland, who introduces Stecher, comes at the question from the supplier’s side. His company, Norm AI, builds an enterprise AI platform for legal and regulatory work. Some of the world’s largest financial institutions use it. So does Norm Law, an affiliated law firm.

Worland opens with a question prompted by the rush to deploy agents. As more companies commit to building them, how does Blackstone decide what it takes to get one right? Stecher begins well before model selection, with the path a piece of information has to travel before an agent can use it.

### Get proprietary and public data out of transactional systems

[1:17](https://www.youtube.com/watch?v=ra8X7_bnujw&t=77s)

Stecher dates a noticeable improvement in usefulness to around February or March of the previous year. Context windows had grown, and models could keep working on a task for significantly longer. People could get more value out of a run. But when he advises portfolio companies that want to build as many agents as possible, the first impediment he looks for is whether an agent can obtain the material the company expects it to work on.

At Blackstone, an asset manager, proprietary information and public information are woven together in the work people do. Some of the relevant material lives in transactional systems. A larger context window helps once the material reaches the model. It does little for a task whose inputs are still scattered across those systems. Stecher wants a cohesive strategy for moving that data somewhere the firm can manage it and from which an agent can retrieve it. The practical question for a proposed agent is whether that path exists for the particular mix of information its job requires.

### Use agents to build straight-through processing and to handle specific decisions

[2:17](https://www.youtube.com/watch?v=ra8X7_bnujw&t=137s)

Stecher gives agents two different jobs in the same application. During development, he has used Factory and other coding tools to help him work through what people actually do and craft software around that workflow. His design objective is still straight-through processing. Where the steps are understood and can run in sequence, he wants code that carries the work through. Looking closely at the workflow also reveals the places where progress has depended on a person making a decision, sometimes a fairly rote one. Those are places to consider putting model intelligence into the running application.

The distinction affects who gets time back. An investment professional or finance professional may have had to stop and make a routine judgment each time a piece of work crossed their desk. An engineer may already know exactly how an application should behave but still have to write the functional test cases, write the performance test cases and bring the libraries underneath the application up to newer versions. Each is a specific piece of work to hand off, with an identifiable person who can use the time for something else.

Dependency upkeep is a particularly personal example for Stecher. He says he has disliked that recurring chore throughout 27 years in engineering. An agent that can go into his source, renovate the code and take on some of the testing has made a material difference to how quickly his team delivers. That experience informs the question he puts back to Worland. Which parts of the work has Norm found it worthwhile to give to an agent?

### How Norm got consistent regulatory outputs in early 2024

[3:46](https://www.youtube.com/watch?v=ra8X7_bnujw&t=226s)

Norm was founded in 2023. Worland remembers the team’s first get-together, a week on Stanford’s campus, as a period when every assumption about model inputs seemed to be changing at once. In his recollection, context windows went from 4K to 8K to 128K, and multimodal capabilities arrived too. He qualifies the chronology himself. What stuck with him was the pace. Then improvements from the frontier seemed to slow for a while, just as Norm needed to turn the models it had into production software.

By early 2024, Norm’s engineers were making decisions explicitly at several layers. They knew certain content would eventually have to enter a model’s context and specified how to get it there. They prescribed the sequence in which models ran. Where a task required consistency checks, they inserted those steps, then distilled the results into an output someone could use. Much of the application’s knowledge about how to complete the job lived in the code coordinating those calls.

Worland likens the result to a large MapReduce problem. The comparison conveys how much work went into organizing individual operations and bringing their results together. He starts to describe this as scaffolding for agents, then corrects himself. The software was much closer to a workflow controlling how models ran. It earned its place by producing accurate results with considerable consistency. For regulatory work, getting a good answer on one run was only part of what Norm needed from the system.

### Tens of thousands of labeled examples made the old workflow testable against an agent

[5:16](https://www.youtube.com/watch?v=ra8X7_bnujw&t=316s)

The size of the workflow created a regression-testing problem. Norm needed a way to tell whether a change had disturbed the quality it had worked so hard to achieve. In building that test capability, it accumulated tens of thousands of examples labeled by experts. Their first purpose was to test the existing implementation. Their lasting value was that they described acceptable work in a form Norm could also use to test a substantially different implementation.

Worland uses a concrete definition of an agent. Give an LLM a set of tools, a goal and some way to verify its work against the goal. Then let it decide how to get there. That changed which decisions Norm had to make in advance. Its workflow had settled the sequence of model calls before a particular task began. The agent could choose how to proceed on the task in front of it. As soon as that approach became viable, Norm could run it against the benchmark it already had.

For the engineers, the result was uncomfortable. They were looking at hundreds of thousands of lines of thoughtfully written code, much of which now seemed obsolete. Some of the upfront structure was actively getting in the way. Worland later says they could get better performance by removing it. He gives the scale of the evaluation set and the codebase, but no numerical uplift. The consequential comparison was between two ways of doing work Norm already knew how to assess.

Worland took the engineering leadership team to upstate New York in a rented Sprinter van after a big snowstorm. At an Airbnb, they asked what platform they would build if they were starting that day. They came away with two requirements for the rebuild. Clients needed agents that could carry out work on their behalf, and Norm’s engineers needed a platform that AI coding tools could help them develop. Worland says the team has been rebuilding the platform to those specifications ever since.

### A model, a harness or a data change can move the answers

[6:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=404s)

Stecher picks up on the asset that made Norm’s comparison possible. In financial services, he wants an evaluation set tied to the actual work the agent performs. Does the output make sense for that work? Is it accurate? Can the firm obtain a consistent, repeatable result? These questions matter after deployment as much as they do when choosing an architecture.

He identifies three changes that can affect answers. First, a model version can change underneath an application. Stecher has seen the answers change with it and the accuracy of the information shift. Second, a change to the harness that runs the model can produce different answers. Third, a change in the data can do the same. Running the evaluations consistently gives the team a measure of the health of the assembled system. A model comparison performed when the team first bought or built the software cannot establish how that software behaves under today’s combination of model, harness and data.

### The bitter lesson reaches the project roadmap

[7:40](https://www.youtube.com/watch?v=ra8X7_bnujw&t=460s)

Worland connects Norm’s experience to Rich Sutton’s bitter lesson, which he dates to 2019. Clever people can invest considerable effort encoding how to solve a problem, only to see an approach that scales with compute eventually outperform their system. He puts the idea in a longer history, from the excitement around Rosenblatt’s perceptron in the 1950s and 1960s to the disputes between proponents of expert systems and connectionist approaches. He recalls early press coverage crediting the perceptron with the prospect of original thought, though he is unsure which publication ran it.

His most pointed example comes from an anecdote about IBM’s translation lab in the 1980s. A lab leader joked that the models improved each time he fired a linguist. The technical issue underneath the joke was where to put effort. Linguists could keep trying to formulate the rules of language, while a statistical translation model could learn from probabilities. Norm had encountered its own version of that choice when removing carefully prescribed steps improved performance. Worland asks what an engineering leader should do with this knowledge when someone brings in a project idea that has to be funded today.

### What would Blackstone gain by shipping six months earlier?

[8:54](https://www.youtube.com/watch?v=ra8X7_bnujw&t=534s)

Stecher starts with the return on the actual request. The request might be to build an agent, put data where it can be used, make a dashboard or develop a trading platform. He works through its value with engineers, product people, the site reliability engineering function and the stakeholders asking for it. On the other side of the comparison is the true cost of building it with the capabilities available today. A belief that enough future compute will make a problem solvable leaves both sides of that investment decision open.

Timing makes the comparison more precise. Suppose a more sophisticated model and sufficient compute are likely to make the problem cheaper to solve in six months. What business value depends on having the application during those six months? The total eventual value of the application may be considerable even when the value of getting it early is small. Blackstone has chosen to defer work when the need was modest, the expected build cost would fall and producing the application immediately offered little additional return.

Six months is Stecher’s example of a decision horizon. He describes making the trade-off regularly, with the urgency and economics of the particular request determining the answer. There is also a cost to repeatedly deciding to wait. If the open question is what someone would actually do with the proposed application, the next model release may arrive without the team having learned anything about demand.

### Build the version a stakeholder can click through

[10:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=624s)

For that question, Stecher shortens the time horizon. Start with a product person or the stakeholder, use a tool such as Fable to work through the concept and plan, then write enough code for someone to use it. His threshold for a workable proof of concept is straightforward. A person can click through it, interact with it and react to what it does. That gives the engineer and stakeholder something more specific to discuss than a PowerPoint describing the proposed software.

Stecher encourages his engineers to get to a proof of concept or proof of value frequently. Spend the tokens, put the result in front of people and iterate while their reactions can still shape it. If the result serves no purpose, discard it. He judges the expense by the value of what the team produces or learns. Counting tokens alone misses the time and money the team can save by finding out early that an idea needs changing or dropping.

This speed depends on the scope of the commitment. The clickable version can exist before cybersecurity review and before the infrastructure work required to run a service at scale in a financial institution. The team can therefore settle a question about usefulness today while leaving a larger production investment for a separate decision.

### Claude Code changed what Worland expected an agent to generalize

[11:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=704s)

Norm’s founders discussed the bitter lesson in 2023, before building the workflow they later reconsidered. Knowing the principle did not tell them how much to spend on each component. By the time Worland faced that choice for Norm’s agent platform, the team had built a harness for running agents, a sandbox for execution and controls over what could leave it. Customers needed those capabilities, and a frontier lab or a third party might release something Norm would want to adopt. Worland came to see investment as a matter of degree, with some parts worth building heavily and others worth keeping easy to replace.

Claude Code prompted him to revise his judgment about generalization. The striking feature for him was the combination of actions available to the model. It could write code, execute that code and use a computer in a way that let it check the result. A developer no longer had to foresee the sequence of operations for every kind of question. Faced with a request outside the distribution of familiar examples, an assistant could spin up a computer and work on the request in the background.

Worland saw that execution-and-verification loop as an ingredient for a much wider range of knowledge work. A consumer could ask an unfamiliar question and let an assistant use a computer to work out the answer. Worland was seeing the same capability in enterprise tools. For Norm, that sharpened the investment question. Agents needed a usable operating environment, and as models could figure out more of the work themselves, Norm had reason to invest in the information it gave them and in measuring what they accomplished with it.

### Norm Law helps score the work a lawyer needs done

[13:45](https://www.youtube.com/watch?v=ra8X7_bnujw&t=825s)

Worland identifies data capture, distillation and making the result available to models as state as areas for sustained investment. Alongside those sit benchmarks and evaluations. Norm Law grew up with the technology company, giving the engineering team direct access to lawyers who can help define what success looks like. That relationship matters when the team has to decide whether a general improvement to the platform has actually improved legal work.

Consider the difference between two roadmap questions. One asks when Norm will deliver an agent that can do three specified things. The other asks how successfully the work gets done across the dimensions a lawyer cares about. For the latter, lawyers and engineers develop a scorecard that can apply to a human completing the task as well as to an agent. The team can follow the scores over time and look for improvements an attorney would recognize in the workflow. It can also change how the agent arrives at a result without treating a departure from the original sequence as a failure. That freedom mattered precisely because the old upfront structure had sometimes reduced performance.

### Production needs traceable software and a human owner for sensitive changes

[14:43](https://www.youtube.com/watch?v=ra8X7_bnujw&t=883s)

Stecher returns to what happens after a promising proof of value. The application has to run inside the infrastructure the customer actually has. Vendors selling AI tools can underestimate how much work falls to the financial institution on the other side of that deployment. Its people have to get the software through cybersecurity review and make it supportable. Stecher wants the software’s behavior to be traceable and to feed into the firm’s SRE platform so the people responsible for the live service can monitor it. Coding agents are getting better at parts of this work, he says, but the gains have been less evident here than in producing prototypes.

Worland describes the ownership rule Norm uses for its core primitives. People oversee the design and implementation, own the code and approve changes. Engineers and agents can build on top of those components. If something built above them needs to change a primitive itself, that change requires human review. The boundary gives developers room to use the platform while keeping a person accountable for changes to the mechanisms protecting sensitive data.

A good demo can make this tension especially clear. Worland describes the experience of seeing an exciting capability and, at the same time, recognizing how much information it would expose if offered with the controls currently available. For Norm’s customers, protecting that information takes precedence over delivering the capability quickly. Worland wants to invest heavily enough in the core to uphold that commitment and more conservatively in layers likely to be superseded. His image for the latter is a conduit through which improved model capabilities can reach the application. An old workflow that insists on directing every step can effectively dam them up.

### Compare the next maintenance cycle with the cost of a replacement

[17:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1044s)

Stecher says that ten years earlier, designing and writing an internal application consumed enough time, money and engineering attention to commit a company to it for a long time. Suggesting a rewrite to the CFO would have seemed absurd. Today he is comfortable building an internal application for a particular purpose and point in time, using it while it serves that purpose and letting it disappear much sooner.

The decision comes due when engineers consider bringing the current version forward again. How much does that round of maintenance cost? How much would a replacement cost to write now, and how much business value might a fresh application deliver? The work already invested in the first version tells the team little about today’s answers. Worland sees a timing problem even within a single year. An assumption about what was expensive in June can still be directing engineering investment in October, after the practical cost has changed considerably.

### A lower cost of scaffolding brings experienced engineers back to the keyboard

[19:16](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1156s)

For Stecher, there is a personal consequence to that change in cost. He spent ten years as a low-level engineer at IBM, using Notepad and vi to write instruction-level code for RISC machines. He still remembers how much he enjoyed it. As his responsibilities grew, getting far enough into an implementation to do useful work became harder. Even starting something that processed data from a stream could mean dealing with enough scaffolding to keep him from getting started at all.

He had retained the ability to architect the system. Coding agents gave him a way to put that knowledge into working software again. He sees the same possibility for engineers with ten, twenty or thirty years of experience. They understand the complexity of system design and can now express more of that understanding directly in code. After a discussion about which software to wait for and which software to discard, that ability to sit down and build again is what Stecher says makes him excited to go to work.

## Key takeaways

- Trace the data a proposed agent needs back to its source systems. Map the workflow separately so ordinary code can carry predictable steps straight through and a model can handle decisions that warrant it.
- Keep expert-labeled evaluations usable across implementations. Rerun them when the model, harness or data changes, and use the same cases to test whether removing workflow steps improves results.
- For a proposed six-month delay, estimate the value of having the application during those six months. When usefulness is uncertain, a clickable proof of concept can resolve that question before a production commitment.
- Give people explicit ownership of sensitive platform primitives. For the internal applications built above them, compare the next round of maintenance with the current cost and value of a replacement.

## Full transcript

[0:12](https://www.youtube.com/watch?v=ra8X7_bnujw&t=12s) I'm Scott Worland. I'm the CTO of Norm AI, and I'm privileged enough to be sharing the stage here with John Stoecker, who's the CTO of Blackstone. Um, from what I understand, John has basically three full-time jobs, it sounds like. You've-

[0:25](https://www.youtube.com/watch?v=ra8X7_bnujw&t=25s) Yep

[0:26](https://www.youtube.com/watch?v=ra8X7_bnujw&t=26s) ... responsible for all of the technology at Blackstone, also advises on their investments, um, and as, as well as serves as a resource for the portfolio companies for all things technology. Did I miss anything?

[0:37](https://www.youtube.com/watch?v=ra8X7_bnujw&t=37s) Nope. You hit it all perfectly, man.

[0:39](https://www.youtube.com/watch?v=ra8X7_bnujw&t=39s) And so-

[0:39](https://www.youtube.com/watch?v=ra8X7_bnujw&t=39s) It's a cool job.

[0:40](https://www.youtube.com/watch?v=ra8X7_bnujw&t=40s) Yeah, and if you're not familiar with what Blackstone does, then you probably have wandered into the wrong conference. Um, so I'm not gonna go into that, but you may not know what Norm does. So really briefly, uh, Norm is an AI company, and we operate in the legal and regulatory space. We have an enterprise AI platform that's deployed in some of the largest financial, uh, institutions in the world, and then we also have an affiliated law firm, Norm Law, and our AI platform also supports that. Um, so since the end of last year, and certainly all of this year,

[1:10](https://www.youtube.com/watch?v=ra8X7_bnujw&t=70s) everyone's been racing to figure out how to deploy AI agents. And I'd love to hear how you're thinking about this at Blackstone, and how you're trying to make sure you get this right.

[1:17](https://www.youtube.com/watch?v=ra8X7_bnujw&t=77s) Yeah, I mean, I think the, the race has definitely been on as, like, the technology has improved mightily, uh, since, you know, really, like, February, March of last year. Like, just in terms of context windows, being able to run stuff for significantly longer amounts of time have, like, resulted in people being able to produce just more value. And one of the things that, like, we've been focused on a lot, and I think every company in our portfolio advise on and work with all the time that they need to have top of mind for, like, being successful with AI agents in general, is you have to get your data story right. And so, like, one of the biggest things,

[1:47](https://www.youtube.com/watch?v=ra8X7_bnujw&t=107s) impediments, and tripping points that I see for companies out there when they want-- they're all in on AI, they wanna build as many agents as possible, they really want to kind of extract value. If they don't have a cohesive data strategy that's moving stuff out of transactional systems into something that can be accessed and easily managed and recalled from, like, an agent perspective, it is going to fail. And so that's been one of the biggest things that we've been focused on. Um, you know, as a big asset manager, we have a ton of data, proprietary data, public data, all woven together. We have to move

[2:17](https://www.youtube.com/watch?v=ra8X7_bnujw&t=137s) that into s- out of systems into something that we can really leverage from an AI perspective. I think the second thing that, you know, has nothing to do with AI, but you can leverage AI to actually figure out how to do the best possible way, is straight-through processing still matters. I'm always shocked at how everybody just talks about, like, agents as being the biggest differentiator in the world. The thing that I've leveraged myself using Factory and a couple other ways to go off and build code inside of the firm is, like, taking time, having agents help to craft something for me, understand the

[2:47](https://www.youtube.com/watch?v=ra8X7_bnujw&t=167s) workflows out there that people are doing, and then make sure that I'm, like, actually writing code to optimize for straight-through processing, and then using agents to inject in intelligence where necessary, where humans were making decisions in the past for a variety of rote work things. So that's really, like, the two key elements that I, I spend a ton of time thinking about every single day. And then the biggest thing from an agent perspective is really stepping back and looking at, like, what is work that we want to build agents to do better, to offload, like, intellectual, like,

[3:16](https://www.youtube.com/watch?v=ra8X7_bnujw&t=196s) cycles from our investment professionals, our finance professionals, our engineers, such that they can do higher value work. And there's a ton of stuff when you think about the software development life cycle, writing test cases, functional test cases, performance test cases, version- versioning forward libraries underneath applications. It's the bane of every engineer's existence. I've hated it for twenty-seven years that I've been doing this. Having an agent be able to go off and do these things for me and renovate my code and my source code has made all the difference in the world for the velocity that we deliver at. So how are you thinking about it at Norm?

[3:46](https://www.youtube.com/watch?v=ra8X7_bnujw&t=226s) Yeah. Um, so Norm was founded in twenty twenty-three, so, you know, that's three or, I guess, four years after GPT-1, but still a year before there was, like, you know, chain of thought. So it was, like, an interesting time in terms of development. I remember when we had our first team get-together, we were at Stanford's campus for a week and, I mean, I'm probably misremembering the exact timeline, but it felt like we went from 4K, uh, to 8K, to a hundred and twenty-eight K, uh, uh, context windows, and then we had multimodal, and everything just felt like it was kinda coming all at once. And then there was a bit of a time where things

[4:16](https://www.youtube.com/watch?v=ra8X7_bnujw&t=256s) did slow down when it was coming from the frontier. And so I think, like a lot of companies in, like, early twenty twenty-four, you know, we were trying to deploy these models as part of, like, our workflow and as part of, like, our production software. And so we ended up adding a lot of structure to the system and, and, and this comes from, uh, you know, all different parts of the stack. So whether that was from, like, how we went and took the content that we knew needed to be, like, uh, eventually added to context to the models, and we had, like, very specific ways in which we did that. Or if there was, like, you know, just the sequencing of the models, if

[4:46](https://www.youtube.com/watch?v=ra8X7_bnujw&t=286s) there was, like, steps we needed to do in terms of looking for consistency, and then ultimately, like, distilling everything down into some useful output. We ended up with, like, something that looked like a large, like, MapReduce problem effectively, right? Whereas, like, we had a lot of scaffolding for the agents, and we were very particular about, like, how these-- well, obviously it's agents, really just how these models ran. It was much more like a workflow. And what this allowed us to do was get, like, really, you know, high quality and also with accurate, um, very accurate, but also just very consistent, which is very important for us in the regulatory space, outputs. Um,

[5:16](https://www.youtube.com/watch?v=ra8X7_bnujw&t=316s) but then one fortuitous thing as part of doing this, 'cause we wanted to have, uh, some way to regression test this big system, we ended up with a really large set of evaluation samples. We had tens of thousands actually of, like, hand-labeled examples by ex- uh, experts. And so as soon as we moved to a place where it was, like, clear that agents were actually a thing, and by agents I simply mean, um, you know, given, like, an LLM a system or giving it a, a set of tools that it can use, give it some goal, and give it some way to verify itself against that goal, allowing it just to go figure out how to do the thing in the best way possible.

[5:47](https://www.youtube.com/watch?v=ra8X7_bnujw&t=347s) We were able to just take this benchmark set and just immediately run on top of it, um, with some of these more agentic approaches. And we saw, you know, I think a lot of people had this moment where you're staring at hundreds of thousands of lines of code your team had really thoughtfully written and realizing most of it was pretty obsolete and was just kind of actually in the way. And so- To update quickly on this, what we did was we, um, we took-- I got the engineering leadership team, we actually rented a, a sprinter van, and we drove to upstate New York. It was actually after a big snowstorm. We had an Airbnb, and it was like, "Forget Norm for a minute.

[6:17](https://www.youtube.com/watch?v=ra8X7_bnujw&t=377s) Let's just say, what would we build today as far as a platform, knowing what we know right now, um, if we were gonna be starting over?" And then from that, we left with basically a set of specs about, like, what this platform needed to be able to do to enable two things. One, deliver agentic experiences to our clients. And then also, um, it needed to be amenable to having AIs, like, co-developers alongside of our own development team, and that's what we've been executing against, uh, ever since then. So I think that's... Did you have a question?

[6:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=404s) No, no, I was gonna say, I think you hit on, like, a key point, Scott, where it says, like, the eval side of the fence that you were talking about before. If you're working in financial services, like, the, the quality bar for what agents have to deliver on is very, very high. And so, I think really thinking through, like, what is your eval set for how you basically test your agents, what they're actually... you know, whatever work they're performing, that the output makes sense, and it's consistent and repeatable. And I've seen across the board, as model versions change underneath the covers, you get different answers.

[7:14](https://www.youtube.com/watch?v=ra8X7_bnujw&t=434s) You get skew in terms of the accuracy of information that comes back. So having good evals so that you can run those consistently and really measure the health of a system is paramount to success. Same thing happens if harnesses change that you're running stuff on top of. You'll get different answers. Data changes, different answers. So it's, it's something where I think, you know, if you're traffic in, in financial services, like, the duty to be accurate is, like, up here, and hitting that is, like, something, like, we think about all the time.

[7:40](https://www.youtube.com/watch?v=ra8X7_bnujw&t=460s) Definitely. I think this leads nicely into this concept that we've been talking about just, like, with the bitter lesson. So, like, everyone's familiar with it, and, you know, there's this idea that these handcrafted, um, systems built by, you know, clever people thinking hard about the problem will oftentimes eventually be dominated by systems that scale with compute, right? And that this is actually... I think Rich Sutton originally talked about this in twenty nineteen, but this has been since the beginning of neural networks. If you go back to, like, the fifties and sixties, like, with Rosenblatt's little perceptron, I think The New York Times ran an article, or somebody did, saying it

[8:10](https://www.youtube.com/watch?v=ra8X7_bnujw&t=490s) was the first, first machine that could ever have its own original thoughts, you know? And so-- And there was actually-- it's-- there's actually quite a, a fascinating history here of how the, you know, expert systems and the connectionist and, um, what ended up happening. A lot of it actually was highly litigated in, in England, and a lot of AI research was dismantled. But one funny thing, if you go back to the eighties even, there was a, a lead at IBM's translation lab, and he made a comment, which is very much the bitter lesson, "Every time I fire a linguist, my models improve," right? You know, the point of having these, like, linguists in there and trying to think about all the rules of language versus, like, training a

[8:40](https://www.youtube.com/watch?v=ra8X7_bnujw&t=520s) statistical model based off just probabilities. And so we speed up to today, and, like, we see this all the time. We've been touching on lots of points about this. And I'm curious, when you think about your roadmaps, when people on your team are bringing you project ideas, like, how do you keep this in mind as you're thinking about this for Blackstone?

[8:54](https://www.youtube.com/watch?v=ra8X7_bnujw&t=534s) Yeah, I mean, a lot of what we do is like, you know, the, the simple equation is, is giving enough compute. Every problem is solvable by, like, hardware and software at some point in time in the future. It's-- Elon talks about this energy-wise, et cetera. So it's like, it's a big map. It's a whole bunch of stuff. How do you bring it back to reality? Like, what I spend a lot of my time doing from, like, my seat working with engineers on my team, with product people, with our SRE function, with, like, stakeholders, is really trying to define, like, you know, what is the ROI of what I'm being asked to

[9:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=564s) work on? And so, like, when I think of building agents, I think about, you know, getting data into the right spot. I think about building dashboards, trading platforms, whatever it is. It's like, if I look at, like, what's the ROI that's gonna be created from the actual ask that's coming in to me from a team perspective, and then, like, what's the, what's the true cost of actually building something today? Like, how can I balance that equation? And then I make a decision of, like, you know, is this something where the models are gonna get good enough, I'm gonna have more than enough compute in six months, that it's not worth trying

[9:54](https://www.youtube.com/watch?v=ra8X7_bnujw&t=594s) to tackle the problem today? It's better to delay it and actually address it with one of the new models. We have made that decision before. It's like, let's actually wait for the model to become slightly more sophisticated so that we can solve these problems with it, because it will cost less at that time, and the need for that application to be built today isn't pressing. There's no, like, real business value increase that can be-- like, will be created because it's produced today versus six months from now. So that's a trade-off that we actually, you know, make regularly and I think about

[10:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=624s) regularly. Um, the other thing that we do is, like, you don't wanna wait too long to basically build something. And so you can always be waiting for the new model, waiting for the new technology or whatever else. Like, the beautiful thing with, like, all of the various different software development harnesses today is if you lean into them, it's amazing how quick you can go from, like, concept, mapping out your planning and concept with, like, Fable for, like, you know, just, "Hey, work with me, and let's define this with the product person or with the stakeholder," to actually writing code, to having, like, a workable proof of concept.

[10:54](https://www.youtube.com/watch?v=ra8X7_bnujw&t=654s) And when I say workable proof of concept, it's something people could click through, interact with, whatever else it is. Doesn't have cybersecurity review, doesn't have all the other things that you need to do in our industry to make sure you, like, put something into production that scales. But you can get to a POC or a POV very quickly. And so, like, that's one big thing I try to encourage all my engineers to do frequently is, like, just get to POC. Don't worry about PowerPoints. Don't worry about anything else. Leverage tokens, which, like, I think it's a fallacy to be, like, counting tokens in some way, shape, or form. It's better to think about producing value. Go off, produce value,

[11:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=684s) produce POVs, put it in front of people, iterate on it quickly, and throw it away if you don't need it. You will spend far less time and money doing those types of things. And so that's, like, a big way that we think about trying to solve it. It doesn't-- maps into the para-- like the, you know, the how do you deal with increasing compute, making it easier to do these things. But, like, I just-- I think at a much shorter timeframe.

[11:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=704s) Yeah. It's so interesting 'cause it's like, you know, I mean, you touched on so many things that I think everybody's probably been thinking about too. And if you go back to twenty twenty-three when we were originally, like, sitting around as a small group talking about what we're building, we were aware of the bitter lesson then. We talked about it. We said, "How do we not get in this trap?" And I think at this point, it's just really evident to me that it's like, it's definitely a gradient, right? It's not this binary thing about you can somehow build the perfect technology to index into all the up, you know, everything coming in from the frontier in a way that you don't ever have to throw things away. So- Really, it gets down to, like, where do we

[12:14](https://www.youtube.com/watch?v=ra8X7_bnujw&t=734s) want to invest heavily and where are we willing to invest conservatively? Because like you said, we need something today, but at the same time, we're just aware that, like, s- some of these investments we make, particularly around things like we've built, you know, fairly complicated systems around all of our agent primitives, so the harness, the sandbox, how egress works, you know, all of these things. And we, we've also want to make sure that as, like, things are coming out either from like a frontier lab or from like a, a third party, that we're able to index into that. And so I think a lot of it is just kind of when do you update? And that- that's one of the hardest

[12:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=764s) things to do. And, like, for me, a good example was I really updated really hard on Cloud Code. Not so much that it was suddenly tools that, like, in my profession that we could use that were really incredible. What it really was for me was just this idea... 'Cause I was a bit skeptical up to that point of how we would get something that was, like, truly generalizable. And it's like, well, what does Cloud Code give you? Well, you can write code, and then it can execute the code and, like, basically run a computer in some verifiable way. At that moment, I was like, well, we have all the ingredients for, like, a truly generalizable intelligence. Because if you can go do this at scale, then,

[13:15](https://www.youtube.com/watch?v=ra8X7_bnujw&t=795s) I mean, you know, of course, like, you can basically do anything in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today. And you fast-forward to now and, like, all these tools that people are using, these, like, AI assistants and things, this is why they're so amazing. It's 'cause you can ask a question it's never seen before, fully out of distribution. It can spin up a computer and go do this thing in the background, and for a consumer, it's amazing. And then we're also seeing this now within the enterprise too. And so for me, that moment was like, well, where are we going to invest heavily? Of course, we still need to invest in what we're building on the agent side. But going back to what,

[13:45](https://www.youtube.com/watch?v=ra8X7_bnujw&t=825s) uh, John mentioned, how we do our data capture and distillation and how we make this available as, like, state to these models, and then also just how we do our benchmarks and evaluations. And so with, like, the law firm, for example, we have this really unique situation where we started this law firm, um, you know, from nothing. So we- we've kind of grown together. They've grown with Norm together. And so we work directly with lawyers to dev- to stand up these benchmarks and evaluations. And effectively, rather than think of a traditional roadmap, like when, when are you gonna build an agent that can go do these three things? It's let's have a scorecard

[14:15](https://www.youtube.com/watch?v=ra8X7_bnujw&t=855s) that determines when an agent or when a human even is ha- success- is successful, um, at completing these tasks across various dimensions. And then we wanna be able to see this just improve over time. So as we make general improvements to the system, we should see an uplift to these, you know, these things like an attorney would expect to improve their workflow, but at the same time not, not getting... kind of going back to the state that we were originally, where we had added a lot of structure to it in a way that ultimately we, we found that we were actually could get even better performance if we were to remove that upfront structure.

[14:43](https://www.youtube.com/watch?v=ra8X7_bnujw&t=883s) Yeah. I think, like, a- another thing that's important to think through is, like, you can produce, like I was saying, a lot more proof of concepts, proof of value very quickly. I still think, like, the hard yards for a lot of development out there is getting it to production, and what I mean by that is, like, infrastructure build-out that actually runs inside of the infrastructure that's available inside of any financial services or enterprise company, uh, for that matter out there. Like, that is a challenge that a lot of folks, like, underestimate. If you're selling AI tools, it's h- that is hard on the other side,

[15:13](https://www.youtube.com/watch?v=ra8X7_bnujw&t=913s) just so you know. I deal with it quite a bit. Uh, the other thing is, is just, like, dealing with, like, the cybersecurity side of the fence and the supportability side of the fence. And so, like, one thing I've noticed, you know, w- the agents are getting better at this now, but it- it's making sure that the actual software that's being written is truly traceable, and it feeds into your SRE platform so that it can be monitorable in production. Those are things still that, like, I like to have people think about. It- it's, it's like they haven't started to scale yet with agents. They will, but it's something that from, uh, my perspective, I, I think about

[15:43](https://www.youtube.com/watch?v=ra8X7_bnujw&t=943s) regularly.

[15:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=944s) Makes sense. And we, our approach to this, and I'm sure a lot of other companies and maybe even you guys have taken this approach as well, is there's parts of our code base that we, like, still require, like, um, complete human oversight from when it comes to designing, building, code ownership, even approving any kind of changes to the code base, and these are, like, our core primitives. And these are things where, you know, basically you can build things on top of these, but the things that you build on top of these are not allowed to, like, change these systems without having, like, human review. And it's, it's because of the things you mentioned, is like, you know, data security for

[16:14](https://www.youtube.com/watch?v=ra8X7_bnujw&t=974s) us is, you know, is, is, is, like, as paramount as delivering quick. It's actually more important. So, and for depends on where you are and if whether it's a consumer product or for us with obviously working with highly sensitive data, like we have to build in so much infrastructure around this and ensure that we're not ever getting out ahead of, like, you know, these parts of the stack, which it can be hard 'cause maybe somebody brings to you, and I'm sure you've seen this, uh, like a really cool demo, and it's really exciting, and you're looking at it terrified  'cause you're like, "This is, this is very cool," but to get to this point in terms of capabilities, like it would expose so much that we're not

[16:44](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1004s) able to do yet. So I think it's just always that trade-off. So we just, you know, basically we invest heavily in these things and then invest conservatively in places where we realize as the frontier continues to improve, um, we wanna make sure we're-- we have, like, conduits. And the way I described it actually this morning with the team is, like, basically throughout our platform, we wanna have, like, conduits for what's coming from the frontier. You don't wanna have dams where, like, the intelligence is coming in and it can't quite get into your stack because you've built things in such a way. Again, it's a gradient. We're gonna get some of it wrong, some of it right, just being very cognizant of it, talking about it a lot, and

[17:14](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1034s) making sure that our core pieces are, are locked down, but then also, you know, being very open about, um, just rip and replace. I think that's, like, a thing that I think everybody's gotten really good at.

[17:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1044s) Yeah, I, I think that's, like, one of the biggest... I mean, like, I've been doing this for twenty-seven years. Sometimes I feel like it's been, like, a hundred years. Other times I feel like it's been five. I think the, the thing that's, like, really made a material difference, like, when I, I look at the ability to write software today versus what I was doing ten years ago, was ten years ago, it took so much time, effort, and money, and people's, like, intellectual capital to produce a piece of software. It was almost like the ship had sailed. Like, by the time you designed it and started writing code, it was out to sea, and you were

[17:54](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1074s) gonna ride that baby for, like, as long as you could until it got to the other side of the ocean. And, like, what I've become very comfortable with today is, like, for the most part, a lot of, like, internal applications can be just thrown away. It's build them, have them work for a purpose and a point in time, but understand that they're going to disappear at some point in time, uh, like now much more quickly than ever before, and then just think about, like, what comes next. And so I, I think one of the challenges with, like, sitting in my seat or any, like, good engineer seat should be constantly questioning, like,

[18:24](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1104s) you know, should I rewrite this, and does it cost me less to rewrite it than it costs me to keep versioning the current version forward? And that is, like, a question that a decade ago never would've came up in anyone's head. If you went to say, "I wanna rewrite an internal application," like, your CFO would've looked at you like you were insane. And I think now it's a completely relevant thing that all of us should be doing 'cause net new applications might produce way more business value at the end of the day. So it's just something I encourage folks to do. It sounds like you guys are doing quite a bit at Norm.

[18:52](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1132s) Yeah, exactly, and I think it's just that concept of what used to be expensive that's now cheap.

[18:56](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1136s) Yep.

[18:56](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1136s) And if you don't update quickly on that, then it's, you know, it, like, you- it's incredible. You said it feels like 27 or 5 years or 100 years or 5 years. It's like that if you over, if you over-index too hard on something that was in June and now it's October, like, you know, it, it feels like in some ways there's been, like, 10 years of technological advancement. So, um, well, great. We only have a j- a, a minute or so left. Is there any closing remarks?

[19:16](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1156s) I mean, I think, like, the, the one thing I'd, like, leave the audience on if, if you want, like, you know, like, good takeaways from this, it's like, you know, we talked about a lot of, like, very interesting things. I think, like, what makes me, like, excited to go to work is, like, it's cool to get back to engineering. One of the things that, like, I value is, like, you know, I was a hardcore low-level engineer for 10 years at IBM and, like, wrote, like, you know, this is way back in the day, of using Notepad and VI to write, like, instruction-level code for RISC machines. It was, like, awesome. It's nerdy as all heck. No one does this stuff anymore, but it

[19:46](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1186s) was, like, fun for me. I still enjoy those days. As you get s- more senior in your career and, like, the barrier to actually writing code on a daily basis disappears, or it, it becomes more, like, hard to get through, like, dealing with a stream, like, processing stuff off a stream. It's just all these things are like, "Oh, my God, I can't even get started 'cause the scaffolding is so hard." With, like, AI, it's been, like, reinvigorating to get back and actually build stuff 'cause it's, like, I never lost my knowledge on how to architect things. I see this with a lot of engineers out there. They

[20:15](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1215s) understand because they've been in the market 10, 20, 30 years. They understand the complexity of system design, which hasn't changed, and now they can actually express that knowledge with these new cool agents. And so I think it's a super exciting time to be, like, starting a company, working at a company, and, like, building software 'cause there's, like, a whole new world that we can create out there. So that's what, like, excites me, so.

[20:35](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1235s) Yeah. Well, cool. Thank you so much. It's been an honor.

[20:37](https://www.youtube.com/watch?v=ra8X7_bnujw&t=1237s) Yep. Thank you, guys, for having us here today. I appreciate it.
