AI Engineer World's Fair 2026
Which AI startups actually land enterprise contracts? — Brian Lewis, Millennium
Read the talk
What makes an AI startup ready for an enterprise contract?
Brian Lewis explains where enterprise AI purchases fail, from pricing and permissions to release controls and data retention, then turns to the foundations buyers must repair themselves.
From a talk by Brian Lewis
At a glance
Ideas worth remembering
Enterprise evaluation starts with the buyer’s problem, working integrations, value-based pricing, and buyer-defined success criteria. Lewis estimates that only about 5% of his demo calls become contracts.
Security requirements must be compatible with normal product operation: retention controls, customer-managed keys, the buyer’s gateway and infrastructure, and access tied to existing entitlement groups.
Operational readiness includes API-accessible administration, configuration audit logs, controlled rollouts, meaningful SLAs, and reachable support engineers.
Data promises need to hold across implementation, feature terms, and subprocessors. Lewis’s examples include retention despite ZDR commitments and beta clauses that permit retention.
Lewis’s explicitly unscientific 60% estimate emphasizes the buyer’s work on architecture, data, integration, and change management. Agents make sound entitlements more urgent because they inherit existing foundations.
The two sides of enterprise AI adoption
After a brief opening apology for an absent speaker, Brian Lewis asks who in the room sells AI products and who buys them. Both groups are represented, and his argument addresses both. Lewis works on product at Millennium, a hedge fund, but explicitly speaks as an individual rather than on behalf of his employer. He limits his examples to evaluating tools rather than describing proprietary systems.
Why teach vendors how to sell to him? Lewis recalls his mother asking that question when he rehearsed the talk. His answer is an interest in how businesses work and how tools become useful. His central hypothesis is that enterprises already leave much of the value available at current model intelligence unrealized. He divides the problem into models and products on the seller’s side, and systems inside the enterprise on the buyer’s side. Better products matter, but their value also depends on the organization’s ability to adopt them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The funnel narrows before a contract is signed
Lewis begins with an internal pain point and a build-or-buy question. For a given problem, he might identify 10 to 15 interesting startups. Initial due diligence reduces those to two or three demo calls, which produce zero or one pilots. Across pilots, he estimates that roughly one in four eventually becomes a contract. His overall estimate is that about 5% of demo calls lead to a signature. These are approximate observations, not a precise conversion model; although he says the result resembles an industry benchmark, the supplied material does not establish that benchmark independently.
The narrowing funnel exposes a disagreement over what enterprise readiness means. A startup may consider its product ready until the buyer walks through actual requirements. Lewis groups the failures into efficacy and commercial value, security, reliability, and legal issues. He attributes 40% of the breakdown to efficacy and commercial issues, without supplying a measurement method or a complete numerical breakdown for the remaining categories. The rest of the talk makes those categories concrete through requirements and unnamed vendor examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prove value within a shrinking pilot window
Efficacy starts with solving the buyer’s actual problem. Lewis adds three conditions: pricing must reflect real value, integrations must be demonstrable on day one, and the buyer defines success criteria. Together, these conditions make a pilot an evaluation against a business need rather than a demonstration of whichever features the vendor wants to showcase. A promised integration cannot establish that the product will work inside the buyer’s environment.
The build-or-buy alternative remains active throughout that evaluation. Lewis describes pitches for products his platform team could rebuild in about six weeks. That does not automatically make purchasing the wrong decision: he explicitly says buying can still be preferable even when rebuilding is possible. It does mean that a vendor needs a persuasive value proposition beyond presenting an idea the buyer can implement.
His pricing example concerns a wrapper whose LLM traffic passed through the buyer’s own gateway. The vendor asked the buyer to report gateway telemetry so it could charge a large margin on that traffic, despite the traffic not running through vendor infrastructure. Lewis says the proposal failed. The objection illustrates a mismatch between what the vendor meters and the value the buyer believes it receives: a usage charge still needs a commercial justification when the buyer supplies the underlying traffic path.
Other failures are simpler: demo promises still lack delivery estimates after two months, or salespeople repeatedly pitch features the buyer already declined. Lewis wants vendors to address the requests they have actually received. The opportunity to do so is becoming shorter. Over his little more than two years at Millennium, he describes pilot expectations moving from six months to three months and potentially to two weeks. That is an observation about accelerating evaluation, not a claim that every deployment must finish in two weeks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Security controls must work with the product
Lewis presents himself as a frontline participant in security discussions, not a security expert. His preferred data arrangement is zero data retention, or ZDR. Where retention is unavoidable, he wants customer-managed encryption keys, with a crucial qualification: enabling them must not break the product. He has encountered vendors that promise compatibility and then discover that their product no longer works. A security capability therefore has to be demonstrated in the configuration the enterprise will actually use.
Deployment requirements follow the same principle. Lewis prefers routing traffic through the buyer’s own gateway and hosting the product in the buyer’s cloud infrastructure. He also calls for SCIM-tied RBAC: the product’s role-based access controls should connect to existing AD groups or other permission and entitlement groups. Different groups need different access at different times, and those decisions should be configurable through an API. A feature that can only be enabled for everyone does not meet that requirement.
For smaller vendors, Lewis also wants at least one real security hire. He acknowledges that security is rarely the first hiring priority, but the buyer needs someone who can understand and discuss the system’s security behavior. This is a staffing requirement alongside the technical controls: the enterprise needs a capable counterpart when questions arise.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where security promises break down
Some failures involve vendors sending data to their own providers or cloud servers despite the buyer’s requirements. Lewis says this has happened during pilots, which use non-production data. Another recurring problem is an integration that appears effortless until the buyer discovers it only functions with unrestricted read and write access. The product may connect successfully, but it cannot operate within the access limits the enterprise requires.
Release defaults can undermine those limits too. Lewis objects to new or beta features turning on automatically with each release; the enterprise needs control over what becomes available. He also describes weekly pilot check-ins where the security architecture diagram is repeatedly promised for the following week. These delays leave the buyer without the evidence needed to understand the system. When a CISO asks what a vendor would do after a breach, an answer that it has never had one does not describe a response plan. Lewis reads that answer as evidence that the vendor does not know how it would respond.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reliability requires control over change
Lewis defines a working control plane through concrete administrative capabilities: every setting should be available through an API, configuration changes should produce audit logs, and the buyer should control their rollout. His example is a system with five administrators. If one changes a setting, intentionally or accidentally, the organization needs to reconstruct what happened. He adds meaningful service-level agreements and a reachable support engineer, making reliability a combination of product controls and operational support.
Rapid release cycles become difficult when individual users control updates across a large deployment. Lewis describes applications shipping multiple times a day and prompting 3,000 people to relaunch. Users then run different versions. If a release breaks SSL certificates, the enterprise struggles to track the failure and deploy consistently at scale. The problem is the interaction between frequent releases, uncontrolled adoption, and poor visibility into the installed versions.
Documentation needs a history as well. Lewis describes support pages that introduce terms or risks absent from the contract, then provide only an updated date rather than a way to compare earlier and later text. The buyer cannot track what changed. He also recounts core APIs being unavailable for multiple hours during a busy trading day, a serious concern in an environment trading billions of dollars. He does not quantify losses from the outage. His requirements are operational visibility and accountability: a status page, an SLA roadmap, and a path to support.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Data commitments must match actual behavior
The legal requirements begin with no training on customer data, regardless of the product or feature involved. Lewis also wants transparency about subprocessors because risks introduced by a vendor’s providers become risks the buyer must manage. He asks for IP indemnification with reasonable liability caps: because the buyer does not control the models, it wants contractual protection if model output infringes intellectual property. These are his purchasing requirements, rather than a statement of what liability the law automatically assigns.
A contractual retention promise can fail in implementation. Lewis recalls a vendor that claimed ZDR, including legally, but later mentioned an observation about customer data it should not have possessed. That conversation exposed a gap between the commitment and actual retention. He describes another pattern in which vendors release new features only as beta, with beta terms permitting data retention. The consequence is that using the new functionality changes the data arrangement. A general assurance about the product is insufficient if the terms attached to individual features allow something different.
Fourth-party risk disclosed on an obscure webpage rather than in the contract creates a related visibility problem. Lewis’s recap brings the requirements together: working security architecture, responsive support engineers, an admin API from the beginning, a 90-day plan for deployment into the buyer’s infrastructure and cloud, and buyer-written success criteria. The 90-day deployment plan addresses a different task from the potentially two-week pilot discussed earlier. What distinguishes a credible vendor is its ability to make the product deployable and governable, with evidence and delivery dates behind the sales pitch.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model progress moves faster than enterprise architecture
Lewis then turns to the buyer’s side. He contrasts his stated average of a new frontier model every 11 days with enterprise architecture that may be a decade or more old. He also contrasts ChatGPT having been available for 43 months at the time of his remarks with companies still working through ERP migrations begun five years earlier. These are comparisons made in the talk, not independently established release statistics. Their purpose is to explain the mismatch in timelines: new capabilities arrive much faster than enterprises can change architecture, security arrangements, and working practices.
His proposed allocation is explicitly unscientific: perhaps 40% of becoming AI native concerns models and products, while 60% concerns data hygiene, clean architecture, integration, enablement, and change management. This is a separate claim from his earlier 40% estimate about purchasing failures. The adoption estimate expresses a priority: model capability alone cannot repair legacy architecture or run an organization’s change process.
Lewis uses a flashlight metaphor to describe AI’s effect on existing systems. It exposes what already works and what does not. Sound foundations can support acceleration; weak foundations cause the application to break down quickly. His recommendation to technology leaders is to address those weaknesses before expecting an AI layer to make the whole environment work. The less glamorous 60% is where he believes adoption should start.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agents inherit permissions, integrations, and knowledge
Entitlements are Lewis’s first example of a foundation that needs attention. People in large enterprises may have too much access or too little, and the way those permissions are managed becomes more difficult when agents act quickly and exercise judgment. The permissions problem is therefore connected to both what an agent can reach and what it can do. Cross-platform integration also becomes more important because an AI system’s usefulness depends on the systems it can reach.
Centralized knowledge is another foundation. Referencing Emil’s earlier keynote, Lewis calls for thinner agents and a smarter substrate. He makes that idea concrete through documentation, support articles, and the knowledge that keeps a company operating: these should be centralized and easy for systems to consume. He also raises the possibility of AI helping write that knowledge in a real-time feedback loop. He describes this as something discussed with vendors, rather than a completed implementation.
Some organizations may need a separate ecosystem for experimentation when the gap between legacy architecture and their intended destination is too large. Lewis presents this as a way to discover what works before deciding how to proceed, and says it is an option they have considered. He does not prescribe a migration design. His closing warning on agents is more direct: they inherit the foundations already in place. He says agents could magnify problems with rogue people or processes by “100X,” a forceful warning rather than a measured multiplier. Fixing entitlements now is his recommended response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What each side must make possible
Lewis closes by placing the vendor requirements in context. He describes Millennium as having about 8,000 people and tight security, compliance, and regulatory requirements. He views it as a demanding customer, though not necessarily the most demanding. His proposition to startups is that designing for customers with this level of control can prepare them to satisfy many others. That is his expectation, not a guarantee of broader market acceptance; the immediate goal is to build a product that such customers can actually use.
Buyers have work of their own: entitlements, governance, and audit logging need sustained attention if AI products are to function inside their systems. Lewis ends with the same motivation that opened the talk—getting things to work better and improving enterprise-grade AI—before thanking the audience. The practical responsibility is shared. Vendors must make their products controllable and supportable, while buyers must build an environment capable of using those controls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:01
[music]
- 0:12
>> Welcome everybody.
- 0:14
Sorry for everybody who was already here
- 0:16
and missed the Coinbase guy. I have no
- 0:17
idea where he went or why he didn't
- 0:19
come. I was actually pretty excited to
- 0:21
hear about his his comments. But today
- 0:24
I'm going to talk about which AI
- 0:25
startups actually win enterprise
- 0:27
contracts.
- 0:28
So to begin
- 0:30
I thought this was going to be a
- 0:31
different audience. I didn't realize
- 0:32
this was going to be mostly people on
- 0:34
the leadership track. I thought I was
- 0:36
going to be speaking to more AI
- 0:37
engineers. So maybe just by show of
- 0:39
hands, how many of you represent like
- 0:41
the engineering or startup or like
- 0:43
seller side?
- 0:45
And then how many of you represent maybe
- 0:46
like the buyer side? Like you're in the
- 0:48
enterprise, you're trying to get these
- 0:49
tools in. Okay, so we got a good mix.
- 0:51
I'm going to try to balance that out
- 0:53
today. I work on product stuff at
- 0:55
Millennium, which is a hedge fund. We
- 0:57
build a lot of stuff. I can't talk about
- 0:59
any of it. So I'm going to talk about
- 1:01
stuff that
- 1:02
we we look at and evaluate. It's going
- 1:04
to be pretty generic. I tried to make it
- 1:06
as interesting as possible while still
- 1:07
getting my compliance department to be
- 1:09
okay with me doing this.
- 1:11
But also need to say legally that I'm
- 1:13
speaking as an individual. I am not
- 1:15
representing my company and all opinions
- 1:17
are my own.
- 1:18
So with that we can dive in.
- 1:21
So I ran through this with my parents
- 1:24
last week. I don't I grew up not that
- 1:26
far from here. And my mom basically
- 1:28
said, "Why are you spending your time
- 1:29
teaching vendors how to sell to you?
- 1:31
Aren't you busy enough already?"
- 1:33
And the real answer to that question is
- 1:36
I really like how stuff works. I like
- 1:38
seeing stuff come together. My
- 1:40
bachelor's degree was in economics. And
- 1:42
I really love seeing things just work
- 1:44
well. So AI's been really interesting
- 1:46
because it's kind of a whole new
- 1:48
paradigm of how businesses are doing
- 1:50
work. That's the whole point of this
- 1:52
track, this AI native enterprise track.
- 1:54
And so even though I don't need more
- 1:57
people DMing me on LinkedIn, um I'm
- 1:59
actually really excited to talk about
- 2:01
this.
- 2:02
So, my hypothesis in short is basically
- 2:05
at current model intelligence, most of
- 2:07
the value available is already being
- 2:09
left on the table. Um this is not a hot
- 2:11
take for most people, I think who work
- 2:13
in enterprise. You've probably seen this
- 2:15
problem. This little stat at the bottom
- 2:17
is uh pretty heavily uh repeated for a
- 2:20
lot of people who work inside of
- 2:21
business circles, and they all kind of
- 2:22
like laugh, and they're like, "Yeah,
- 2:24
yeah, you know, all these AI tools, how
- 2:26
much are they actually doing?" Um and I
- 2:28
want to talk about why. So, there's kind
- 2:31
of two sides to this becoming gen AI
- 2:33
native. Um you have models and products,
- 2:35
which are one side, that's the seller
- 2:36
side, and then you also have systems and
- 2:39
all of what's inside of the enterprise,
- 2:40
that's the buyer side. So, that's the
- 2:41
side that I deal with a lot.
- 2:44
Um so, we're going to talk first about
- 2:45
the seller side, and then we're going to
- 2:46
talk about the buyer side. So, per pain
- 2:48
point, uh a lot of my job is kind of go
- 2:50
around the company and figure out like
- 2:52
what are the pain points? Uh what are we
- 2:53
trying to solve for? Uh can we buy it?
- 2:55
Can we build it?
- 2:57
So, let's say for a given pain point,
- 3:00
maybe I identify 10 to 15 startups that
- 3:02
look really interesting. Like, "Huh,
- 3:04
maybe these guys can solve our problem
- 3:05
for us, we don't have to build it."
- 3:07
Um of those, after doing a little bit of
- 3:09
due diligence on my own, I might
- 3:11
schedule two to three demo calls.
- 3:14
Of those, we probably will land zero or
- 3:17
one pilots.
- 3:19
And of those,
- 3:21
probably one in four of those longer
- 3:23
term will actually end up with a
- 3:24
contract.
- 3:25
So, what does this mean? This means
- 3:26
about 5% of all of our demo calls
- 3:29
actually end up in a signed contract. Um
- 3:31
and this tracks with the industry. I had
- 3:33
no idea that this was actually a
- 3:34
benchmark, um but it turns out that
- 3:37
there's quite a bit out there that
- 3:38
indicates that this is really similar
- 3:40
across the board.
- 3:43
So, I want to talk about what enterprise
- 3:44
ready actually means from the inside, uh
- 3:47
because we have a lot of startups that
- 3:48
tell me what enterprise ready means, and
- 3:50
And we go through all of our
- 3:51
requirements, and then we have a very
- 3:52
different idea of what enterprise ready
- 3:54
actually means. Um so we're going to
- 3:56
talk about what breaks down and why.
- 3:59
So 40% of this is efficacy, so just
- 4:02
value, uh commercial issues.
- 4:04
Then there's a lot that dies in
- 4:06
security. There's other things that die
- 4:08
in reliability, and then there's some
- 4:09
stuff that dies in legal.
- 4:12
So we're going to start with the
- 4:13
requirements that we put forward and
- 4:14
then some of the things that we've seen
- 4:15
go wrong across various AI companies
- 4:18
that we work with.
- 4:19
So our requirements for efficacy, maybe
- 4:21
unsurprisingly, the product actually
- 4:23
needs to solve the problem.
- 4:24
Um that seems pretty clear, but that's
- 4:27
not always super clear.
- 4:29
Uh the next one is pricing models that
- 4:31
need to reflect real value.
- 4:33
Clear demonstration of integrations on
- 4:35
day one, not a hypothetical.
- 4:38
And we define the success criteria, not
- 4:41
the vendor.
- 4:42
So things we've seen go wrong, uh
- 4:44
vaporware in short. Um we've had a lot
- 4:46
of startups who come in, they pitch us
- 4:48
an idea, uh and it's something that our
- 4:50
platform team can rebuild in about 6
- 4:51
weeks. So this is not a knock, uh this
- 4:54
is actually just what's going on in the
- 4:56
industry everywhere, um on all sides of
- 4:58
the equation. Um sometimes it's actually
- 5:01
better for us to build, and sometimes it
- 5:02
is still better for us to buy even if we
- 5:03
could rebuild.
- 5:06
Upside down pricing, so this one's
- 5:07
crazy. Um
- 5:09
We had a startup just recently tell us,
- 5:11
"Hey, um we know that all of the LLM
- 5:13
traffic that we're using for our wrapper
- 5:16
is passing through your LLM gateway, but
- 5:19
we want you to report your gateway
- 5:21
telemetry to us so that we can then
- 5:23
price a huge margin on top of that, even
- 5:26
though none of it's running through our
- 5:27
infrastructure."
- 5:29
Um
- 5:30
that did not work.
- 5:31
Another one is promises in demo calls,
- 5:33
but no ETAs after 2 months. Um this is
- 5:35
pretty common. Um
- 5:37
not a lot to say here.
- 5:40
Um and then repitching features we've
- 5:41
already declined. So
- 5:43
if you're a salesperson, um
- 5:45
my best advice to you is listen to your
- 5:47
customers. It's not novel, but uh it
- 5:49
still seems to be a struggle for some.
- 5:51
Uh it's really just better to address
- 5:52
the things that we've asked for. So, the
- 5:54
other thing I want to point out at the
- 5:55
very bottom of this slide is the pilot
- 5:57
window collapsing. So,
- 5:59
uh I've been at Millennium for a little
- 6:00
over 2 years, and when I started, a lot
- 6:03
of these pilot timelines that people
- 6:04
were used to were like, "Oh, maybe we'll
- 6:05
run a pilot for 6 months." And then, not
- 6:08
that long after that, it was like, "Oh,
- 6:10
maybe we only need it for 3 months." And
- 6:12
anymore, it's like, "Maybe we can do
- 6:14
this pilot for 2 weeks." Uh because it's
- 6:16
just accelerated so rapidly.
- 6:20
Um so, then moving on to security. Uh
- 6:21
this is a huge one. I'm not a security
- 6:23
expert, but I do run kind of frontline
- 6:25
defense on talking to a lot of startups
- 6:27
about security. And so, these are a lot
- 6:28
of the things that that come up over and
- 6:30
over.
- 6:31
Uh ZDR. So, this is a really hot topic.
- 6:33
Obviously, a lot going on with Fable, uh
- 6:36
mandatory data retention requirements,
- 6:38
uh and then a whole other battleground
- 6:40
around customer-managed encryption keys.
- 6:42
So, ZDR is always best, of course. If
- 6:45
that's not possible, customer-managed
- 6:47
encryption keys
- 6:49
and, with a big parentheses, that don't
- 6:50
break the product. Um there are a lot of
- 6:52
things that people are like, "Oh, yeah,
- 6:54
it's fine. It'll work with
- 6:55
customer-managed encryption keys."
- 6:56
And then, it breaks the product. Uh so,
- 6:58
that's a big product uh issue that we
- 7:00
have to work through with people.
- 7:03
Other requirements, bring your own
- 7:04
gateway. We prefer to route all of our
- 7:06
own traffic through our own gateway and
- 7:08
BYO infrastructure. Uh we would prefer
- 7:10
to host it in our own cloud
- 7:11
infrastructure and have something that's
- 7:12
deployable in our systems.
- 7:15
This is another really big one. Um
- 7:17
SCIM-tied RBAC. So, for all of you who
- 7:20
who get that jargon, um
- 7:22
it's really important that we can tie
- 7:24
our AD groups or other permission and
- 7:26
entitlement groups to role-based access
- 7:28
control. We want to make sure that we
- 7:30
don't just turn on features for
- 7:31
everybody across the board. A lot of
- 7:33
people don't think about this when
- 7:34
they're designing their systems. They're
- 7:35
like, "Oh, this is a great feature. We
- 7:37
should just turn it on for everybody."
- 7:39
Um when you work at a a
- 7:40
enterprise, that's not something that
- 7:42
people want to do. Um there are usually
- 7:44
different groups who should have
- 7:45
different access at different times, and
- 7:47
most of all we want it to be
- 7:48
configurable via API.
- 7:51
Um
- 7:52
for smaller companies, we want to see at
- 7:54
least one real security hire. So, this
- 7:57
is something that's really important. We
- 7:58
know that security is not the first
- 8:00
thing that people hire for. Um but in
- 8:02
the age of AI, this is a very real
- 8:03
problem, and we need to make sure that
- 8:05
the startups we're working with actually
- 8:06
have somebody who can understand what's
- 8:08
going on from the security standpoint.
- 8:11
Uh so, some of the things we've seen go
- 8:12
wrong,
- 8:13
um
- 8:14
outright people just sending data to
- 8:16
their vendors, uh cloud servers, and not
- 8:19
following
- 8:20
any of what we've asked for. Um this has
- 8:22
been a problem in pilots. Uh thankfully,
- 8:24
all of our pilots run non-production
- 8:26
data.
- 8:28
Another one, like we kind of talked
- 8:29
about, um read write all default scopes.
- 8:32
So, there's a lot of really cool tools
- 8:34
out there, integrations, features.
- 8:36
They're really flashy. You can click a
- 8:38
button, and it'll integrate with
- 8:40
everything. And then you get a little
- 8:41
bit deeper and find out the only way
- 8:43
that it'll work is if you literally give
- 8:45
it read write all to everything, uh
- 8:47
which is a huge problem.
- 8:49
Another one, uh kind of along the same
- 8:51
lines, all or new beta features on by
- 8:53
default with each release. So, if you're
- 8:55
an enterprise, you don't want everything
- 8:57
just turned on with each release.
- 9:00
Um so, being able to control that,
- 9:02
and then
- 9:03
the line that we hear a lot, which is
- 9:05
we'll get you the security architecture
- 9:07
diagram next week.
- 9:08
Uh we do weekly check-in calls during a
- 9:10
pilot, and then we hear this over and
- 9:11
over. Uh it's not usually a great sign.
- 9:17
Uh the question that we often have our
- 9:19
CISO end up asking, which is um
- 9:22
what are you going to do if there's a
- 9:23
breach?
- 9:24
And we get this response, well, we
- 9:25
haven't had a breach yet.
- 9:27
Uh
- 9:28
with the subtext of we don't know what
- 9:29
we would do if we did.
- 9:31
Um okay, reliability, this is another
- 9:33
one. So, a control plane that actually
- 9:35
works.
- 9:36
We want to see every admin setting
- 9:37
available via API.
- 9:39
We want to see audit logs on config
- 9:41
changes. So, if there are five different
- 9:43
people who are given admin access and
- 9:45
somebody accidentally changes something
- 9:47
or does it because uh maybe it was
- 9:49
really late at night and maybe they had
- 9:51
too many drinks. Uh we actually want to
- 9:52
see what happened. Uh we want to be able
- 9:54
to control the rollout on these changes.
- 9:57
Uh we want to see real SLAs and a
- 9:59
reachable support engineer. That goes a
- 10:00
very, very long way.
- 10:02
So, uh things we've seen go wrong,
- 10:05
a lot of apps that are rapidly
- 10:06
prototyping, they're shipping so quickly
- 10:08
that they are maybe shipping updates
- 10:10
multiple times a day and there's a
- 10:11
really attractive little button that
- 10:13
says relaunch to update and it happens
- 10:15
across 3,000 people. We have no way of
- 10:17
tracking what's going wrong. Maybe then
- 10:19
like SSL certificates break in one of
- 10:21
the new releases and then we have no way
- 10:22
of tracking because everybody's on a
- 10:24
different version
- 10:25
um and we have no way of being able to
- 10:26
deploy at scale. Um that's really
- 10:28
challenging.
- 10:30
No documentation versioning.
- 10:32
So, support articles with new terms or
- 10:34
risks that are not actually in the legal
- 10:36
contract but show up in the website
- 10:38
somewhere in a random support page and
- 10:39
then we have no way of tracking what
- 10:40
they were before versus after and it
- 10:42
just says updated yesterday.
- 10:44
All of these are real examples, by the
- 10:46
way. I am not naming and shaming. Um I'm
- 10:48
just shaming. So,
- 10:50
uh maybe if any of you are familiar, you
- 10:52
can put it together. Um core API's down
- 10:55
for multiple hours during a busy trading
- 10:57
day. Uh that is a really big problem for
- 10:59
us because we run production systems. We
- 11:02
are trading billions of dollars.
- 11:04
Um this is a really big issue for us.
- 11:07
And then lastly, no SLA roadmap or
- 11:09
status page. Um the status page is a big
- 11:11
one.
- 11:12
Okay, last, legal issues. So, we don't
- 11:15
want anybody training on our data
- 11:17
regardless of what type of feature or
- 11:18
product it is.
- 11:20
Uh we also want to see a lot of
- 11:21
transparency in the sub processors.
- 11:24
Um any fourth-party risk becomes our
- 11:26
risk.
- 11:28
We want to see IP indemnification with
- 11:30
reasonable liability caps. Uh we do not
- 11:32
control the models, so if there's output
- 11:34
that is IP infringing, we don't want to
- 11:35
be held liable for it.
- 11:37
So, we have seen in pilots that people
- 11:39
claim they have ZDR, they have it
- 11:41
legally, but then they find out or we
- 11:43
find out later that they actually retain
- 11:45
some of our data because they say, "Hey,
- 11:47
we were looking at something and we
- 11:48
noticed this thing." And we're like,
- 11:49
"How did you notice that? You weren't
- 11:50
supposed to have this data." And they're
- 11:52
like, "Oh, yeah, you're right."
- 11:54
Um so, that's not great. If you say ZDR,
- 11:57
do ZDR. Um
- 11:59
next, every feature that is conveniently
- 12:01
beta with permissive data retention
- 12:03
clauses. So, we've seen some vendors who
- 12:06
they will stop releasing new features in
- 12:09
general availability. They will only
- 12:11
make them beta, and then the beta comes
- 12:13
with a secret little clause that says
- 12:15
that they're allowed to retain our data,
- 12:16
which is a very sneaky way of trying to
- 12:18
get our data. We don't like that. Um not
- 12:20
great.
- 12:22
Another one kind of similar is
- 12:23
fourth-party risk that's tucked away on
- 12:25
a random website page that's not listed
- 12:28
in the contract. Uh this is a really big
- 12:30
problem for us managing risk.
- 12:34
So, um
- 12:35
it was the best of times, it was the
- 12:36
worst of times. As a recap, the best
- 12:38
startups have security architecture that
- 12:40
actually works, support engineers who
- 12:42
respond, an admin API from the
- 12:44
beginning, a 90-day plan that deploys
- 12:47
into our infrastructure and cloud, and
- 12:49
success criteria that we write.
- 12:51
The worst AI startups don't have any
- 12:52
security architecture diagrams, no path
- 12:54
to a support engineer, no deployment
- 12:56
control or audit logs, no ETAs,
- 12:59
and salesmanship over solid product
- 13:01
building.
- 13:02
Um this is really just kind of a recap
- 13:04
of like what I have been through over
- 13:07
the last 2 years. Um I actually don't
- 13:08
think that any of this is novel,
- 13:10
um but it is codifying a lot of what I
- 13:12
feel like is good and best practice.
- 13:15
Um okay, so a new frontier model comes
- 13:17
out on average every 11 days, but your
- 13:19
architecture might be a decade or more
- 13:21
old. So, you've got a bunch of cool new
- 13:23
models, there's some amazing
- 13:24
capabilities is there, and then you have
- 13:26
profitability on the other side of it.
- 13:28
And what's in the middle? Maybe it's
- 13:29
your legacy architecture, probably a lot
- 13:31
of security and privacy issues, and a
- 13:33
lot of change management. Um ChatGPT has
- 13:36
only been out for 43 months. There are a
- 13:38
lot of companies who are still doing an
- 13:39
ERP migration that might have been from
- 13:41
5 years ago. Um so, the timelines are
- 13:43
very asymmetric. Uh and I think that
- 13:46
sometimes we forget about that.
- 13:49
Um okay. So, my thesis again, half or
- 13:52
more of getting to AI native is unsexy
- 13:54
and has absolutely nothing to do with
- 13:56
AI.
- 13:57
Um AI models and products today can't
- 13:59
fix your legacy architecture. Although,
- 14:01
if any of you are startup people, that's
- 14:02
a great one to go for.
- 14:04
Um and it also can't run your change
- 14:06
management. These are unscientific
- 14:08
numbers that I'm putting up here, but I
- 14:10
hypothesize that 40% of getting to AI
- 14:12
native is AI models and products. The
- 14:15
other 60% is all the other stuff that no
- 14:16
one really likes talking about anymore,
- 14:18
uh which is like data hygiene, clean
- 14:20
architecture, having good integration,
- 14:22
strong enablement, and change
- 14:23
management.
- 14:26
Um I really look at AI as a flashlight,
- 14:28
not a band-aid. Um I really think that
- 14:30
AI shines a light on a lot of what's
- 14:33
already working or not working. It can
- 14:34
accelerate what's working really well,
- 14:36
and it breaks down very quickly when
- 14:38
things don't work well.
- 14:39
Um I don't think that it's a band-aid,
- 14:41
and I think that for everybody who's in
- 14:42
tech leadership, it's really important
- 14:43
to remember that if you have issues in
- 14:46
your technology estate, those need to be
- 14:48
addressed before trying to plug in AI
- 14:51
and just having everything rip. Um it's
- 14:54
it's not going to work.
- 14:55
Um so, hehehe again, maybe an unpopular
- 14:58
message, but I really believe that we
- 15:00
all need to start with the boring 60%. I
- 15:03
think that's where we all need to start
- 15:04
to get to the other side of the road.
- 15:07
So,
- 15:08
what did we learn as we shine the
- 15:10
flashlight internally?
- 15:12
Again, not revealing anything super
- 15:13
proprietary, but I do think these are
- 15:15
big picture lessons.
- 15:16
Number one, entitlements. Entitlements
- 15:18
need a new paradigm.
- 15:20
Uh there are a lot of people in a lot of
- 15:22
large enterprises who are over entitled,
- 15:24
under entitled. The entitlements model
- 15:26
and how it works and how it's managed,
- 15:28
all of that breaks down when you think
- 15:30
about agents and how quickly you want
- 15:31
agents to work and what you want them to
- 15:33
work on and their ability to exercise
- 15:34
judgment. Um the entire paradigm just
- 15:37
shifts.
- 15:38
Another one is cross-platform
- 15:40
integration moved up the stack. So, AI
- 15:42
is only as good as what it reaches and
- 15:44
we want it everywhere. Uh so, having
- 15:46
things that can integrate across
- 15:48
platforms is really important uh even
- 15:50
more than it already was.
- 15:53
Another one is centralized knowledge.
- 15:54
So, this is something that um Emil
- 15:57
brought up this morning in his keynote,
- 15:59
which is that basically we need thinner
- 16:01
agents and a smarter substrate. Um
- 16:03
centralized knowledge is really key to
- 16:04
that. So, all of your documentation, all
- 16:06
your support articles, everything that's
- 16:07
going on inside of your company that's
- 16:09
making it work, um all of that needs to
- 16:11
be centralized and easily consumable.
- 16:13
Even better if AI can help write that in
- 16:16
real time in a feedback loop. Uh that's
- 16:18
something we've been talking about with
- 16:19
some of our vendors.
- 16:21
Another one is a separate ecosystem for
- 16:22
experimentation. Um some companies may
- 16:25
need to get here.
- 16:27
That gap between your legacy
- 16:28
architecture and where you want to go
- 16:29
might be so vast that you actually just
- 16:31
decide, "Hey, maybe we need a separate
- 16:33
ecosystem to do a lot of this work,
- 16:35
figure out what does work and what
- 16:37
doesn't and then kind of go from there."
- 16:39
Um and that's something that we thought
- 16:40
about as well.
- 16:42
So, to just put a finer point on the
- 16:44
agents and the entitlement thing,
- 16:46
uh agents inherit your foundations. So,
- 16:48
I strongly recommend that everybody fix
- 16:51
their entitlements if they are not
- 16:52
working really well now
- 16:54
um because this is something that if you
- 16:56
think about the problems that you run
- 16:57
into when things go rogue, processes go
- 16:59
rogue, people go rogue, agents are going
- 17:01
to like 100X that problem. Um so, this
- 17:04
is really something that's worth
- 17:05
figuring out now.
- 17:08
So, to kind of recap uh as I wrap up
- 17:11
here, the recipe, if you are one of the
- 17:13
people in the first half who are raising
- 17:15
your hand on like, "What do I need to do
- 17:17
if I'm a startup and I want to work with
- 17:18
a really difficult large customer?
- 17:20
Millennium's got like 8,000 people. We
- 17:23
have very, very tight security,
- 17:24
compliance, regulatory requirements.
- 17:27
Um this is the stuff that we care about.
- 17:30
And we want to see more startups doing
- 17:33
work that allows us to work with them.
- 17:35
Um I really view this as like one of the
- 17:38
highest bars. We're probably not the
- 17:39
highest, um although we're probably
- 17:41
pretty close.
- 17:43
Um and I think if you can architect your
- 17:44
startup to work with companies like this
- 17:46
with this kind of architecture, um
- 17:48
you're probably going to be able to
- 17:49
satisfy basically everybody else.
- 17:52
Um on the other side
- 17:54
for anybody who's buying, uh these are
- 17:56
the things that I think again that kind
- 17:57
of that boring 60% that really deserves
- 18:00
a lot of work. Um off entitlements,
- 18:03
governance, audit logging, etc. Um these
- 18:05
are the things that I think we need to
- 18:06
have in terms of systems to get it
- 18:09
working on the other side of the
- 18:10
equation.
- 18:12
So,
- 18:13
that's it. Um
- 18:15
my only motivation here is to getting
- 18:17
stuff working better and having better
- 18:19
enterprise grade AI.
- 18:21
Uh that's a QR code to my LinkedIn and I
- 18:23
appreciate all of your time. Thank you.
- 18:26
>> [applause]
- 18:40
[music]