# Which AI startups actually land enterprise contracts? — Brian Lewis, Millennium

Brian Lewis, Millennium

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/7A65O-0lvKE-which-ai-startups-actually-land-enterprise) · [Video](https://www.youtube.com/watch?v=7A65O-0lvKE)

## Which AI startups actually land enterprise contracts?

Brian Lewis explains why promising demos fail enterprise evaluation, then turns to the permissions, integrations, knowledge and change management that buyers must repair before agents can work well.

### A useful model still needs somewhere to work

[0:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=12s)

Enterprise AI already leaves much of the value available at current model intelligence unused. That is the opening hypothesis of [Brian Lewis](https://ai.engineer/speakers/brian-lewis), who works on product at Millennium, a hedge fund, and speaks here personally rather than for his employer. His examples concern evaluating outside products, rather than describing proprietary systems.

The room contains both sellers and buyers, and both have work to do. Sellers supply models and products. Buyers supply the systems those products must enter. The talk follows those two sides in order: first the requirements that eliminate vendors, then the organizational foundations that buyers need to improve.

Why teach vendors how to sell to him? Lewis’s mother asked essentially that question when he rehearsed the talk. His answer is an interest in seeing things come together and work well. The purchasing advice that follows is unusually concrete: enterprise readiness means making a product usable inside the customer’s organization, through the awkward details that a demo can skip.

### The funnel starts with a problem, then tests the purchase

[2:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=162s)

The buying process begins with an internal pain point and a build-or-buy question. For a given problem, Lewis might find 10 to 15 interesting startups. Initial due diligence reduces those to two or three demo calls, which lead to zero or one pilots. Roughly one in four pilots eventually becomes a contract. His overall estimate is that about 5% of demo calls end in a signature. These are approximate observations; the resemblance to an industry benchmark is his reported comparison.

What eliminates the others? The failures fall into efficacy and commercial value, security, reliability and legal issues. Lewis assigns 40% to efficacy and commercial issues, without developing a measurement method. Walking through the requirements exposes the distance between a startup’s idea of enterprise readiness and a purchase the customer can actually approve.

At [4:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=259s), efficacy becomes four practical conditions:

- **Solve the problem:** The product must address the pain point that started the evaluation.
- **Price the value:** The commercial model must make sense for what the buyer receives.
- **Show working integrations:** Integration needs to be demonstrable on day one. A future possibility cannot establish that the product works in the customer’s environment.
- **Use buyer-defined success:** The customer writes the success criteria, keeping the evaluation tied to its need.

The internal alternative remains live. Some pitches describe something the platform team could rebuild in about six weeks. That does not settle the decision: buying can still be preferable even when rebuilding is possible. It does require the vendor to offer a persuasive reason to purchase beyond the existence of an implementable idea.

One pricing proposal makes the mismatch tangible. A vendor’s wrapper used LLM traffic passing through the buyer’s own gateway. None of that traffic ran through vendor infrastructure. The vendor nevertheless asked the buyer to report its gateway telemetry so the vendor could charge a large margin on the traffic. The proposal failed. The traffic path supplied a usage number, but the buyer did not accept that number as justification for the proposed markup.

Other failures consume the evaluation window without moving the product forward: promises still lack delivery estimates after two months, or salespeople keep pitching features the customer already declined. Meanwhile, pilot expectations have shortened over Lewis’s little more than two years at Millennium—from six months, to three months, to potentially two weeks. A shorter pilot gives future promises less room to compensate for missing functionality; the pilot window describes evaluation, rather than the time needed for a full deployment.

### Security controls have to preserve the product’s usefulness

[6:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=373s)

Security evaluation asks whether the product works under the buyer’s required configuration. Lewis describes himself as a frontline participant in these conversations rather than a security expert. His preferred arrangement is zero data retention, or ZDR. Where retention cannot be avoided, he wants customer-managed encryption keys—with the crucial qualification that enabling them must not break the product. A promised security capability is little help if the application stops functioning when the customer uses it.

The deployment and access requirements address different kinds of control:

- **Traffic and hosting:** Route LLM traffic through the buyer’s gateway and support deployment in the buyer’s own cloud infrastructure.
- **Group-based permissions:** Tie role-based access control, or RBAC, to existing AD groups or other entitlement groups through the SCIM-tied arrangement Lewis requests. The practical requirement is that the product use the organization’s existing group structure to give different groups different access at different times.
- **API configuration:** Make access and feature decisions configurable through an API, rather than offering only an all-users switch.
- **Security staffing:** Even a smaller vendor should have at least one real security hire who can understand and discuss the system’s behavior.

A one-click integration can conceal a serious restriction: it may only work if granted read and write access to everything. The connection succeeds technically, while the access model makes it unsuitable for the enterprise. New or beta features enabled automatically with each release create a related problem. The buyer needs to decide which capabilities become available, to whom and when.

Other pilot failures concern the data path itself: vendors send data to their providers or cloud servers despite the buyer’s requirements. Lewis says the pilots use non-production data. Repeatedly postponing the security architecture diagram leaves the buyer unable to understand that path. And answering a breach-response question with the fact that no breach has happened yet supplies no response plan. Prior good fortune cannot explain what happens next.

### A fast release cycle needs a controllable rollout

[9:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=557s)

Reliability begins with a working control plane: the administrative part of the product that lets the customer configure and manage it. Every admin setting should be available through an API, configuration changes should produce audit logs, and the customer should control rollout. If five administrators have access and one changes something late at night, the organization needs to reconstruct what happened. Meaningful service-level agreements, or SLAs, and a reachable support engineer complete the operating arrangement.

Consider the update example at [10:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=602s). An application ships updates multiple times a day and offers 3,000 users an attractive “Relaunch to update” button. Each user can move independently, so the deployment becomes a population running different versions. If a new release breaks SSL certificates, the customer struggles to track the failure because users no longer share a consistent version. Frequent shipping, independent adoption and poor version visibility combine to make deployment at scale difficult.

How does a convenient update button become an operational problem? The diagram follows the change from rapid releases to mixed versions to difficult diagnosis. The branch matters: a faulty release reaches a deployment whose users have updated at different times. Rollout control would let the customer manage that transition; audit logs would help reconstruct configuration changes alongside it.

Documentation also needs a history. A support page can introduce terms or risks missing from the contract, then display only “Updated yesterday.” Without earlier versions, the customer cannot compare what changed. Operational visibility matters just as much during outages: Lewis recounts a core API being unavailable for multiple hours during a busy trading day in an environment trading billions of dollars, without attributing a financial loss to that outage. A status page, an SLA roadmap and access to support give the buyer ways to understand and manage disruption.

Independent updates fragment a deployment
A release problem becomes harder to trace when thousands of users independently adopt different versions.
- Updates multiple times a day — The application ships rapidly.
- 3,000 users can relaunch — Individual users choose when to update.
- Different installed versions — The deployment loses version consistency.
- A release breaks SSL certificates — Certificate failure illustrates a possible release problem.
- Difficult diagnosis and deployment — The customer struggles to track failures and deploy consistently at scale.
- Updates multiple times a day → 3,000 users can relaunch (Offers updates)
- 3,000 users can relaunch → Different installed versions (Independent adoption)
- Updates multiple times a day → A release breaks SSL certificates (A new release may fail)
- Different installed versions → Difficult diagnosis and deployment (Makes failures hard to track)
- A release breaks SSL certificates → Difficult diagnosis and deployment (Requires investigation)

### Data commitments must survive implementation and feature terms

[10:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=647s)

The legal requirements cover how data is used, who handles it and who bears risk. No training on customer data should apply regardless of the feature or product. Subprocessors—the vendor’s own providers—need to be transparent because their risks become risks the customer must manage. Lewis also wants intellectual-property indemnification with reasonable liability caps, giving the buyer contractual protection against infringing model output when it does not control the models. These are purchasing requirements, rather than a statement of how the law automatically assigns liability.

A ZDR promise can exist in the agreement and still fail in the application. During one pilot, a vendor mentioned something it had noticed in customer data. That observation exposed possession of information the vendor was not supposed to retain. The conversation changed the buyer’s understanding of the product: a legal commitment had not produced the promised data behavior. Lewis’s instruction is appropriately short: “If you say ZDR, do ZDR.”

Feature terms can change the arrangement too. Lewis describes vendors releasing new features only as beta, with beta clauses permitting data retention. Using the new functionality then introduces retention permission that the customer may not expect from the general product assurance. Fourth-party risk tucked away on a webpage rather than listed in the contract creates another visibility gap. The customer needs to understand the data arrangement across implementation, individual features and outside providers.

The credible vendor package now comes into focus: working security architecture, responsive support engineers, an admin API from the beginning, a 90-day plan for deployment into the buyer’s infrastructure and cloud, and buyer-written success criteria. The deployment plan addresses a different task from the potentially two-week pilot discussed earlier. Both require concrete work and dates behind the pitch.

### Model progress outruns enterprise change

[13:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=784s)

The talk turns to buyers at [13:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=795s). Lewis contrasts his stated average of a new frontier model every 11 days with architecture that may be a decade or more old. Companies can still be working through ERP migrations begun five years earlier. The release cadence is his comparison, rather than an independently established statistic here; its purpose is to expose the mismatch between rapidly arriving capabilities and slowly changing enterprise systems.

His allocation of adoption work is explicitly unscientific: 40% models and products, 60% data hygiene, clean architecture, integration, enablement and change management. This is a separate estimate from the earlier 40% assigned to purchasing failures. It expresses where he thinks effort belongs. Model capability does not itself repair legacy architecture or carry an organization through a change in working practices.

The flashlight metaphor explains the mechanism. AI exposes the systems and practices already helping or hindering work. It can accelerate a process whose foundations work well, while quickly encountering weaknesses in one that does not. Technology leaders therefore need to address those weaknesses before expecting an AI layer to make everything work. That is the reason to start with Lewis’s “boring sixty percent.”

### Agents inherit permissions, reach and organizational knowledge

[14:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=897s)

Entitlements—the access a person or system is allowed to exercise—are the first foundation to revisit. People in large organizations may already have too much access or too little. The way those permissions are managed becomes more difficult when agents act quickly, work on a wider set of tasks and exercise judgment. Permission design has to account for both what an agent can reach and what it can do there.

Cross-platform integration comes next. AI’s usefulness depends on the systems it reaches, so connections across platforms become more important when the intended workflow spans them. A capable agent confined to one disconnected application cannot draw on the rest of the organization’s systems.

Centralized knowledge supplies another foundation. Drawing on Emil’s earlier keynote, Lewis calls for “thinner agents and a smarter substrate.” He makes that framing concrete: documentation, support articles and the knowledge that keeps a company operating should be centralized and easy to consume. The agent can then draw on shared organizational knowledge. A possible next step is for AI to help write that knowledge in real time, feeding new information back into the shared base. Lewis describes vendor discussions about this feedback loop, rather than a completed implementation.

Some organizations may also need a separate ecosystem for experimentation. When the gap between legacy architecture and the intended destination is too large, that environment could help teams discover what works before deciding how to proceed. Lewis presents it as an option they have considered, without prescribing a migration design.

The closing warning returns to entitlements: agents inherit the foundations already in place. Integration supplies reach, knowledge supplies usable context, and permissions determine which actions that reach allows. Existing weaknesses come along too. Lewis’s warning that agents could multiply rogue behavior by 100 times is rhetorical rather than a measured multiplier; his practical recommendation is to fix entitlements now, before faster execution magnifies the problem.

Lewis closes with a demanding customer in view: Millennium has about 8,000 people and tight security, compliance and regulatory requirements. He expects startups designed for customers with this level of control to be well prepared for many other buyers, though that is an expectation rather than a guarantee of acceptance. The immediate opportunity is simpler: build products that such customers can actually use.

Buyers must make that use possible too. Authentication, entitlements, governance and audit logging deserve sustained work alongside product selection. Vendors need to supply controllable, supportable software; enterprises need systems that can apply those controls. Getting enterprise-grade AI to work requires both sides of the table to finish their part.

## Key takeaways

- Evaluate against the buyer’s problem, working integrations, value-based pricing and buyer-written success criteria. Internal reproducibility keeps the build-or-buy decision active.
- Security controls must work in the configuration the customer will use: retention limits, customer-managed keys, customer infrastructure and gateway, and permissions tied to existing groups.
- Frequent releases require rollout control and version visibility. API administration, configuration audit logs, meaningful SLAs and reachable support make the product manageable after the demo.
- Data promises must hold across actual retention behavior, beta-feature terms and subprocessors. A general assurance can fail at any of those points.
- Agents inherit the enterprise’s permissions, integrations and knowledge. Lewis’s informal 60% allocation puts those foundations and change management at the center of adoption work.

## Resources

- [Brian Lewis on LinkedIn](https://www.linkedin.com/in/brianthomaslewis/) — The contact destination Lewis offers at the end for following up on enterprise AI adoption and vendor evaluation.

## Related talks

- [Thinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer](https://ai.engineer/talks/VGN22pPpb-8-thinner-agents-on-smarter-substrate-ontology-based) — Explore the shared knowledge substrate that Lewis connects to thinner agents.
- [CIAM for AI: Authn/Authz for Agents — Michael Grinich, CEO of WorkOS](https://ai.engineer/talks/D4Dswf-__RM-ciam-ai-authn-authz-agents) — Continue the identity and access-control questions raised by agents inheriting enterprise entitlements.
- [Agents Need Feature Flags](https://ai.engineer/talks/zU4EagB311U-agents-need-feature-flags) — Pairs with the need to control feature activation and rollout across different groups of users.

## Slides

### Which AI Startups Actually Win Enterprise Contracts?

Brian Lewis shares lessons from 100+ AI startup evaluations over 24 months, during which a handful of contracts were signed. He speaks as an individual.

[0:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=23s) [Title slide introducing Brian Lewis’s lessons from evaluating AI startups at Millennium Management.](https://assets.ai.engineer/sites/aiecode2025/sha256/16a9e993a125874e4cbba7a5326d02e5e80ace3ddd2c2179088ad4a20264aa08.webp)

Which AI Startups Actually Win Enterprise Contracts?

AI ENGINEER WORLD'S FAIR 2026

Lessons from evaluating 100+ AI startups at Millennium Management
Brian Lewis · AI Product Lead, Millennium Management

24 months · 100+ evaluations · a handful of signed contracts

Speaking as an individual. Views are my own and do not represent Millennium Management or its affiliates. All content is general; no specific commercial agreements or proprietary processes are discussed.

### Why Teach Vendors How to Sell?

Lewis answers his mother’s question: he enjoys seeing how things work and come together, and finds AI interesting because it is changing how businesses work.

[1:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=83s) [A quotation from the speaker’s mother asks why he spends time teaching vendors how to sell to him.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4ee5014157ec7d6b09702289c6fa37120bebb759a888fe1fd0e51b9631bbec8.webp)

“Why are you spending your time teaching vendors how to sell to you? Aren't you busy?

— my mom, last week

### AI Value Left on the Table

Lewis argues that AI value depends on effective models and products and enterprise systems ready to receive them. He cites MIT NANDA’s 2025 finding that 95% of enterprise GenAI pilots show no measurable P&L impact.

[2:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=120s) [The slide argues that much available AI value is unused and cites a 2025 MIT NANDA figure that 95% of enterprise GenAI pilots show no measurable profit-and-loss impact.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ac37ac33c42a99d75912913772f5a7d4a9db550fa45b88936818fcf85ef5a29.webp)

At current model intelligence, most of the value available is already being left on the table.

Getting to AI-native has two sides: the models and products have to work well — and your systems have to be ready to receive them.

95% of enterprise GenAI pilots show no measurable P&L impact
MIT NANDA · The GenAI Divide: State of AI in Business, 2025

### The Funnel per Pain Point

Per pain point, the funnel narrows from 10–15 identified vendors to 2–3 demo calls and 0–1 pilots. About 5% of demo calls end in a signed contract, matching the slide’s cited MIT NANDA benchmark.

[3:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=205s) [A funnel shows 10–15 vendors identified, 2–3 demo calls, 0–1 pilots and approximately 0.25 contracts per pain point. A callout says about 5% of demo calls end in a signed contract.](https://assets.ai.engineer/sites/aiecode2025/sha256/708c8fe93a4b65924dc2818971078e8d9f6a19d757d98df919e598aa703a81df.webp)

THE FUNNEL · PER PAIN POINT
≈50 vendors screened a year

Identified: 10–15
Demo calls: 2–3
Pilots: 0–1
Contracts: ~0.25

~5%
of demo calls end in a signed contract.

MIT's industry-wide success rate: also ~5%. — NANDA, 2025

### What enterprise-ready means

Over the last 24 months, efficacy accounted for 40% of failed evaluations, followed by security at 30%, reliability at 20%, and legal at 10%.

[3:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=221s) [Overview of failed evaluations by efficacy, security, reliability, and legal causes.](https://assets.ai.engineer/sites/aiecode2025/sha256/8de5a350feaf89fa3d41ba3c93a093b79840113085dd10bf9d0a5089e9c1aef2.webp)

What “enterprise-ready” actually means from the inside.

Four categories. Each has non-negotiables, negotiables, and red flags.

Where failed evaluations die — share by cause, last 24 months
40% Efficacy
30% Security
20% Reliability
10% Legal

### Efficacy — Issues

Lewis says products must solve the problem. Efficacy accounts for approximately 40% of failed evaluations, while the slide shows pilot windows shrinking from six months to 2–4 weeks.

[4:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=250s) [Efficacy requirements begin with solving the problem; a timeline shows the shrinking pilot window.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e42a26fcb9eec0f5268ed9fa36027b3d8b088566a4d4939ff9d24ac2b4976f5.webp)

Efficacy — Issues
01 / 04
≈40% of failed evaluations die here

Our requirements
▪ The product needs to solve the problem, effectively

Things we've seen go wrong

The pilot window, collapsing: 6 months → 90 days → 2–4 weeks

### Efficacy — Issues

Millennium expects effective products and pricing that reflects value. Lewis cites a vendor seeking margin on traffic through Millennium’s own gateway. Pilot windows have shrunk from 6 months to 2–4 weeks.

[5:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=304s) [Efficacy requirements and examples of product and pricing problems, with a shrinking pilot window.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c032df57c5339433aada4c5738f55bd92110f4f44b32ad2b782bc84068e0564.webp)

01 / 04
Efficacy — Issues
≈40% of failed evaluations die here

Our requirements
▪ The product needs to solve the problem, effectively
▪ Pricing models that reflect real value
▪ Clear demonstration of integrations on day one
▪ We define success criteria — not them

Things we've seen go wrong
× Vaporware. (Our platform team rebuilt in six weeks)
× Upside-down pricing: XX% margin charged on traffic routed through our own gateway

The pilot window, collapsing: 6 months → 90 days → 2–4 weeks

### Security — Issues

≈30% of failed evaluations end at security. Lewis asks for control over data, infrastructure and access, and flags repeated delays on security diagrams and unclear breach response.

[9:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=555s) [Security requirements and examples involving data handling, permissions, defaults, documentation, and incident response.](https://assets.ai.engineer/sites/aiecode2025/sha256/c49200406c8597385aaac824568b0ba865b84d8e868059a1cce203c15eb492ce.webp)

Security — Issues
02 / 04
≈30% of failed evaluations die here

Our requirements
▪ ZDR — or CMEK (that doesn't break the product!)
▪ BYO gateway. BYO infrastructure.
▪ SCIM-tied RBAC, configurable via API
▪ At least one real security hire

Things we've seen go wrong
× Sending data to the vendor's cloud servers
× “Read.Write.All” default scopes
× All new/beta features "on" by default with each release
× "We'll get you the security architecture diagram next week"
× "Well, we haven't had a breach yet [but definitely not sure what we would do if we did]"

### Reliability — Issues

Lewis wants API-accessible admin controls, audit logs and controlled rollouts. Update prompts across 3,000 seats can leave users on different versions; he also cites an API outage lasting hours during a busy trading day.

[9:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=570s) [Reliability requirements and an example of a burdensome update process.](https://assets.ai.engineer/sites/aiecode2025/sha256/19124663d0606a75012cb566256a046658ae7107e3ef31c50c81e1fbb6ae903b.webp)

03 / 04
Reliability — Issues
≈20% of failed evaluations die here

Our requirements
▪ A control plane that actually works
▪ Every admin setting available via API
▪ Audit logs on config changes; we control rollout
▪ Real SLAs. A reachable support engineer.

Things we've seen go wrong
× “Relaunch to update” × 3,000 seats

[11:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=665s) [Reliability requirements alongside examples involving updates, documentation, API outages and missing service information.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9f87a39166946bc31b1a71235e61101ab86fc2de183f36236384908ff24869c.webp)

03 / 04
Reliability — Issues
≈20% of failed evaluations die here

Our requirements
▪ A control plane that actually works
▪ Every admin setting available via API
▪ Audit logs on config changes; we control rollout
▪ Real SLAs. A reachable support engineer.

Things we've seen go wrong
× “Relaunch to update” × 3,000 seats
× No documentation versioning. Support articles with new terms/risks and simply say "Updated yesterday"
× Core APIs down for multiple hours during a busy trading day
× No SLA, roadmap, or status page

### Legal — Issues

Legal issues account for approximately 10% of failed evaluations. Lewis flags retention clauses in beta features and subprocessors omitted from contracts as risks alongside requirements for no data training and IP indemnification.

[12:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=741s) [Legal requirements for data use, subprocessors and indemnification, alongside examples of retention and fourth-party risks.](https://assets.ai.engineer/sites/aiecode2025/sha256/73244b6217ff627175146af833c4dd84d6e715eebff2c0f862fb655f9f991e36.webp)

04 / 04
Legal — Issues
≈10% of failed evaluations die here

Our requirements
▪ No training on our data — “preview” and “beta” included
▪ Transparent subprocessors — 4th-party risk becomes our risk
▪ IP indemnification with reasonable liability caps

Things we've seen go wrong
× ZDR in the contract, retention in practice
× Every feature is conveniently “beta” with permissive data retention clauses
× 4th-party risks quietly tucked in to obscure pages on their website, but subprocessors missing from contract

### The best and worst AI startups

Lewis contrasts startups that provide working security, responsive support and a 90-day deployment plan with those missing basic controls, timelines and engineering access.

[12:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=768s) [A comparison of AI startups across security, support, administration, deployment planning and product practices.](https://assets.ai.engineer/sites/aiecode2025/sha256/d80bf18ad73a4a22bfd0743e5040d3d18ea1809226346bc0ea36792b5d512c47.webp)

It was the best of times, it was the worst of times.

+ The best AI startups
Security architecture that actually works
Support engineers who respond when everything breaks
Admin API from the start
A 90-day plan that deploys into our infra / cloud
Success criteria that we wrote

− The worst AI startups
No security architecture diagrams
No path to a support engineer
No deployment control or audit logs
No ETAs
Salesmanship over solid product building

### Where the value leaks

Legacy architecture, security and privacy, and change management stand between new models and the P&L. The slide contrasts 43 months since ChatGPT’s release with a 60-month ERP migration to illustrate the timeline gap.

[13:14](https://www.youtube.com/watch?v=7A65O-0lvKE&t=794s) [Diagram places legacy architecture, security and privacy, and change management between models and profit. Bars compare 43 months since ChatGPT's release with a 60-month ERP migration.](https://assets.ai.engineer/sites/aiecode2025/sha256/38cff842475600a95deee09b86825fec2bb1f88d59e128db32e84e909b73822c.webp)

WHERE THE VALUE LEAKS

A new frontier model comes out every 11 days, but your architecture may be a decade (or more) old.

Models
Legacy architecture
Security & privacy
Change management
Your P&L

43 months since ChatGPT first released
60 months since you started your current ERP migration

### The work of becoming AI-native

Lewis’s unscientific hypothesis assigns 60% of AI-native effort and funding to data hygiene, architecture, integration, enablement and change management, and 40% to models and products.

[13:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=827s) [A donut chart assigns 40% of the effort to AI models and products and 60% to other work, including data hygiene, architecture, integration, enablement and change management.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ffadb638ef7d9f2a4c3c5c3cbd695270ac9c125fbcd467c831363b1c96a86a1.webp)

Half (or more) of getting to "AI-native" is unsexy, and has nothing to do with AI.

AI models and products today can't fix your legacy architecture or run your change management.

Effort and funding needed for your AI-native outcome:
40% New AI models and products
60% Everything else
data hygiene · clean architecture · integration · enablement · change management

Gartner, 2026: half of GenAI projects die after proof of concept — data quality, risk controls, costs, unclear value. All buyer-side.

### Start with the boring 60%

Lewis argues that AI exposes existing strengths and weaknesses. His diagram places legacy architecture, security and privacy, and change management between models and P&L. He calls this foundational work the “boring 60%,” a hypothesis.

[14:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=865s) [The slide says AI is a flashlight, not a bandage, and urges starting with the foundational work. A diagram shows architecture, security and change management between models and profit.](https://assets.ai.engineer/sites/aiecode2025/sha256/7de663a422349077182e6e3d932b07a8b153aba66abaed3300181333ee52b471.webp)

AI is a flashlight, not a bandaid.
Start with the boring 60%.

Models
Legacy architecture
Security & privacy
Change management
P&L

Brian Lewis · AI Product Lead, Millennium Management · AI Engineer World's Fair 2026

### What the flashlight found

After 100+ vendor evaluations, Lewis identifies access, integration and centralized documentation as foundations, alongside a separate ecosystem for experiments away from production trading systems.

[16:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=980s) [Four findings from vendor evaluations: a new entitlement paradigm, cross-platform integration, centralized knowledge and a separate experimentation ecosystem.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6fc895dd3ff5239cb6027ab52075bea5661ba495cd01d046a6535e0c046ff42.webp)

What the flashlight found.

Four things 100+ vendor evaluations exposed about our own estate.

1 Entitlements needed a new paradigm
Human-era access models didn't survive contact with AI tools.

2 Cross-platform integration moved up the stack
AI is only as good as what reaches. We want it everywhere.

3 Centralized knowledge
We overhauled how documentation is produced, written, and consumed — docs as code, AI-friendly by default.

4 A separate ecosystem for experimentation
We want to try nearly everything — none of it touching production trading systems.

“We need thinner agents with a smarter substrate.” — Emil Eifrem, Neo4j

### Agents inherit your foundations

Lewis urges companies to fix entitlement leaks before agents act across their estate. He suggests agents could multiply the identity and entitlement problem 100×.

[16:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1001s) [Agents inherit existing entitlements; the slide urges fixing them before agents multiply the problem.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d9d206275b050232f16c1f464b54b62f3659ac109d8e46a54571e7f65c8a727.webp)

Agents inherit your foundations.
Invest in fixing entitlements now.

If entitlements leak with humans in the loop, imagine thousands of agents at machine speed.

100× the identity & entitlement problem, once agents can act across your estate

### The recipe for startups

Lewis’s checklist for selling to a demanding enterprise spans data controls and bring-your-own infrastructure, plus API administration, five 9's, responsive support, and customer requests merged in under six months.

[17:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1027s) [Nine startup requirements covering security, infrastructure, administration, reliability, support, customer requests and pricing.](https://assets.ai.engineer/sites/aiecode2025/sha256/db7b071ff7d1a2a6ca7b38b1ae12390c2d8c08fb75cb0ebc1dbf37d4af8d338d.webp)

The recipe — for startups.
Side one — the models and products have to work well

01 ZDR everywhere possible; CMEK on everything else
02 BYO-infra and BYO-gateway
03 ≥ 1 real security staff
04 Functioning SCIM, RBAC, and clean admin panel
05 Every config setting manageable via API
06 Five 9's
07 Support engineers that respond
08 Customer FR's that have merged in <6 months
09 Scalable pricing

### The other recipe for you

Buyers also need to prepare their own systems. Lewis identifies authentication, governance, audit logging, cost controls, observability, and fallback systems as work needed to make enterprise AI function.

[17:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1072s) [Eight system requirements include authentication, entitlements, governance, audit logging, privacy handling, cost controls, observability and fallback systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/a648e4b90ef5a05e3391765f18b647ac6e7d41aecdd7b05ac67b75d1d84bab9e.webp)

The other recipe — for you.
Side two — your systems have to receive them well

01 Auth
02 Entitlements
03 Data governance
04 Audit logging
05 PII handling
06 Cost controls
07 Observability
08 Gateway / fallback systems

### AI Engineer World's Fair

[18:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1114s) [AI Engineer World's Fair event card on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d72c0ac9134d2a002d661832903f922752e10d4ec32d305ef72e3dd0ad4be72.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=12s) >> Welcome everybody.

[0:14](https://www.youtube.com/watch?v=7A65O-0lvKE&t=14s) Sorry for everybody who was already here

[0:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=16s) and missed the Coinbase guy. I have no

[0:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=17s) idea where he went or why he didn't

[0:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=19s) come. I was actually pretty excited to

[0:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=21s) hear about his his comments. But today

[0:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=24s) I'm going to talk about which AI

[0:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=25s) startups actually win enterprise

[0:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=27s) contracts.

[0:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=28s) So to begin

[0:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=30s) I thought this was going to be a

[0:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=31s) different audience. I didn't realize

[0:32](https://www.youtube.com/watch?v=7A65O-0lvKE&t=32s) this was going to be mostly people on

[0:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=34s) the leadership track. I thought I was

[0:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=36s) going to be speaking to more AI

[0:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=37s) engineers. So maybe just by show of

[0:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=39s) hands, how many of you represent like

[0:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=41s) the engineering or startup or like

[0:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=43s) seller side?

[0:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=45s) And then how many of you represent maybe

[0:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=46s) like the buyer side? Like you're in the

[0:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=48s) enterprise, you're trying to get these

[0:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=49s) tools in. Okay, so we got a good mix.

[0:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=51s) I'm going to try to balance that out

[0:53](https://www.youtube.com/watch?v=7A65O-0lvKE&t=53s) today. I work on product stuff at

[0:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=55s) Millennium, which is a hedge fund. We

[0:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=57s) build a lot of stuff. I can't talk about

[0:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=59s) any of it. So I'm going to talk about

[1:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=61s) stuff that

[1:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=62s) we we look at and evaluate. It's going

[1:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=64s) to be pretty generic. I tried to make it

[1:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=66s) as interesting as possible while still

[1:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=67s) getting my compliance department to be

[1:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=69s) okay with me doing this.

[1:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=71s) But also need to say legally that I'm

[1:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=73s) speaking as an individual. I am not

[1:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=75s) representing my company and all opinions

[1:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=77s) are my own.

[1:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=78s) So with that we can dive in.

[1:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=81s) So I ran through this with my parents

[1:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=84s) last week. I don't I grew up not that

[1:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=86s) far from here. And my mom basically

[1:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=88s) said, "Why are you spending your time

[1:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=89s) teaching vendors how to sell to you?

[1:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=91s) Aren't you busy enough already?"

[1:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=93s) And the real answer to that question is

[1:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=96s) I really like how stuff works. I like

[1:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=98s) seeing stuff come together. My

[1:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=100s) bachelor's degree was in economics. And

[1:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=102s) I really love seeing things just work

[1:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=104s) well. So AI's been really interesting

[1:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=106s) because it's kind of a whole new

[1:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=108s) paradigm of how businesses are doing

[1:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=110s) work. That's the whole point of this

[1:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=112s) track, this AI native enterprise track.

[1:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=114s) And so even though I don't need more

[1:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=117s) people DMing me on LinkedIn, um I'm

[1:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=119s) actually really excited to talk about

[2:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=121s) this.

[2:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=122s) So, my hypothesis in short is basically

[2:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=125s) at current model intelligence, most of

[2:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=127s) the value available is already being

[2:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=129s) left on the table. Um this is not a hot

[2:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=131s) take for most people, I think who work

[2:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=133s) in enterprise. You've probably seen this

[2:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=135s) problem. This little stat at the bottom

[2:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=137s) is uh pretty heavily uh repeated for a

[2:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=140s) lot of people who work inside of

[2:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=141s) business circles, and they all kind of

[2:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=142s) like laugh, and they're like, "Yeah,

[2:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=144s) yeah, you know, all these AI tools, how

[2:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=146s) much are they actually doing?" Um and I

[2:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=148s) want to talk about why. So, there's kind

[2:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=151s) of two sides to this becoming gen AI

[2:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=153s) native. Um you have models and products,

[2:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=155s) which are one side, that's the seller

[2:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=156s) side, and then you also have systems and

[2:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=159s) all of what's inside of the enterprise,

[2:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=160s) that's the buyer side. So, that's the

[2:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=161s) side that I deal with a lot.

[2:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=164s) Um so, we're going to talk first about

[2:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=165s) the seller side, and then we're going to

[2:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=166s) talk about the buyer side. So, per pain

[2:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=168s) point, uh a lot of my job is kind of go

[2:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=170s) around the company and figure out like

[2:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=172s) what are the pain points? Uh what are we

[2:53](https://www.youtube.com/watch?v=7A65O-0lvKE&t=173s) trying to solve for? Uh can we buy it?

[2:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=175s) Can we build it?

[2:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=177s) So, let's say for a given pain point,

[3:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=180s) maybe I identify 10 to 15 startups that

[3:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=182s) look really interesting. Like, "Huh,

[3:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=184s) maybe these guys can solve our problem

[3:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=185s) for us, we don't have to build it."

[3:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=187s) Um of those, after doing a little bit of

[3:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=189s) due diligence on my own, I might

[3:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=191s) schedule two to three demo calls.

[3:14](https://www.youtube.com/watch?v=7A65O-0lvKE&t=194s) Of those, we probably will land zero or

[3:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=197s) one pilots.

[3:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=199s) And of those,

[3:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=201s) probably one in four of those longer

[3:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=203s) term will actually end up with a

[3:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=204s) contract.

[3:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=205s) So, what does this mean? This means

[3:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=206s) about 5% of all of our demo calls

[3:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=209s) actually end up in a signed contract. Um

[3:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=211s) and this tracks with the industry. I had

[3:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=213s) no idea that this was actually a

[3:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=214s) benchmark, um but it turns out that

[3:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=217s) there's quite a bit out there that

[3:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=218s) indicates that this is really similar

[3:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=220s) across the board.

[3:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=223s) So, I want to talk about what enterprise

[3:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=224s) ready actually means from the inside, uh

[3:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=227s) because we have a lot of startups that

[3:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=228s) tell me what enterprise ready means, and

[3:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=230s) And we go through all of our

[3:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=231s) requirements, and then we have a very

[3:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=232s) different idea of what enterprise ready

[3:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=234s) actually means. Um so we're going to

[3:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=236s) talk about what breaks down and why.

[3:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=239s) So 40% of this is efficacy, so just

[4:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=242s) value, uh commercial issues.

[4:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=244s) Then there's a lot that dies in

[4:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=246s) security. There's other things that die

[4:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=248s) in reliability, and then there's some

[4:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=249s) stuff that dies in legal.

[4:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=252s) So we're going to start with the

[4:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=253s) requirements that we put forward and

[4:14](https://www.youtube.com/watch?v=7A65O-0lvKE&t=254s) then some of the things that we've seen

[4:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=255s) go wrong across various AI companies

[4:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=258s) that we work with.

[4:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=259s) So our requirements for efficacy, maybe

[4:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=261s) unsurprisingly, the product actually

[4:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=263s) needs to solve the problem.

[4:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=264s) Um that seems pretty clear, but that's

[4:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=267s) not always super clear.

[4:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=269s) Uh the next one is pricing models that

[4:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=271s) need to reflect real value.

[4:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=273s) Clear demonstration of integrations on

[4:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=275s) day one, not a hypothetical.

[4:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=278s) And we define the success criteria, not

[4:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=281s) the vendor.

[4:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=282s) So things we've seen go wrong, uh

[4:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=284s) vaporware in short. Um we've had a lot

[4:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=286s) of startups who come in, they pitch us

[4:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=288s) an idea, uh and it's something that our

[4:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=290s) platform team can rebuild in about 6

[4:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=291s) weeks. So this is not a knock, uh this

[4:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=294s) is actually just what's going on in the

[4:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=296s) industry everywhere, um on all sides of

[4:58](https://www.youtube.com/watch?v=7A65O-0lvKE&t=298s) the equation. Um sometimes it's actually

[5:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=301s) better for us to build, and sometimes it

[5:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=302s) is still better for us to buy even if we

[5:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=303s) could rebuild.

[5:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=306s) Upside down pricing, so this one's

[5:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=307s) crazy. Um

[5:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=309s) We had a startup just recently tell us,

[5:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=311s) "Hey, um we know that all of the LLM

[5:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=313s) traffic that we're using for our wrapper

[5:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=316s) is passing through your LLM gateway, but

[5:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=319s) we want you to report your gateway

[5:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=321s) telemetry to us so that we can then

[5:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=323s) price a huge margin on top of that, even

[5:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=326s) though none of it's running through our

[5:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=327s) infrastructure."

[5:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=329s) Um

[5:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=330s) that did not work.

[5:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=331s) Another one is promises in demo calls,

[5:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=333s) but no ETAs after 2 months. Um this is

[5:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=335s) pretty common. Um

[5:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=337s) not a lot to say here.

[5:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=340s) Um and then repitching features we've

[5:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=341s) already declined. So

[5:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=343s) if you're a salesperson, um

[5:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=345s) my best advice to you is listen to your

[5:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=347s) customers. It's not novel, but uh it

[5:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=349s) still seems to be a struggle for some.

[5:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=351s) Uh it's really just better to address

[5:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=352s) the things that we've asked for. So, the

[5:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=354s) other thing I want to point out at the

[5:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=355s) very bottom of this slide is the pilot

[5:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=357s) window collapsing. So,

[5:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=359s) uh I've been at Millennium for a little

[6:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=360s) over 2 years, and when I started, a lot

[6:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=363s) of these pilot timelines that people

[6:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=364s) were used to were like, "Oh, maybe we'll

[6:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=365s) run a pilot for 6 months." And then, not

[6:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=368s) that long after that, it was like, "Oh,

[6:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=370s) maybe we only need it for 3 months." And

[6:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=372s) anymore, it's like, "Maybe we can do

[6:14](https://www.youtube.com/watch?v=7A65O-0lvKE&t=374s) this pilot for 2 weeks." Uh because it's

[6:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=376s) just accelerated so rapidly.

[6:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=380s) Um so, then moving on to security. Uh

[6:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=381s) this is a huge one. I'm not a security

[6:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=383s) expert, but I do run kind of frontline

[6:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=385s) defense on talking to a lot of startups

[6:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=387s) about security. And so, these are a lot

[6:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=388s) of the things that that come up over and

[6:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=390s) over.

[6:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=391s) Uh ZDR. So, this is a really hot topic.

[6:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=393s) Obviously, a lot going on with Fable, uh

[6:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=396s) mandatory data retention requirements,

[6:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=398s) uh and then a whole other battleground

[6:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=400s) around customer-managed encryption keys.

[6:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=402s) So, ZDR is always best, of course. If

[6:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=405s) that's not possible, customer-managed

[6:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=407s) encryption keys

[6:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=409s) and, with a big parentheses, that don't

[6:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=410s) break the product. Um there are a lot of

[6:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=412s) things that people are like, "Oh, yeah,

[6:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=414s) it's fine. It'll work with

[6:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=415s) customer-managed encryption keys."

[6:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=416s) And then, it breaks the product. Uh so,

[6:58](https://www.youtube.com/watch?v=7A65O-0lvKE&t=418s) that's a big product uh issue that we

[7:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=420s) have to work through with people.

[7:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=423s) Other requirements, bring your own

[7:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=424s) gateway. We prefer to route all of our

[7:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=426s) own traffic through our own gateway and

[7:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=428s) BYO infrastructure. Uh we would prefer

[7:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=430s) to host it in our own cloud

[7:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=431s) infrastructure and have something that's

[7:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=432s) deployable in our systems.

[7:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=435s) This is another really big one. Um

[7:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=437s) SCIM-tied RBAC. So, for all of you who

[7:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=440s) who get that jargon, um

[7:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=442s) it's really important that we can tie

[7:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=444s) our AD groups or other permission and

[7:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=446s) entitlement groups to role-based access

[7:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=448s) control. We want to make sure that we

[7:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=450s) don't just turn on features for

[7:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=451s) everybody across the board. A lot of

[7:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=453s) people don't think about this when

[7:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=454s) they're designing their systems. They're

[7:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=455s) like, "Oh, this is a great feature. We

[7:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=457s) should just turn it on for everybody."

[7:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=459s) Um when you work at a a

[7:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=460s) enterprise, that's not something that

[7:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=462s) people want to do. Um there are usually

[7:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=464s) different groups who should have

[7:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=465s) different access at different times, and

[7:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=467s) most of all we want it to be

[7:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=468s) configurable via API.

[7:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=471s) Um

[7:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=472s) for smaller companies, we want to see at

[7:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=474s) least one real security hire. So, this

[7:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=477s) is something that's really important. We

[7:58](https://www.youtube.com/watch?v=7A65O-0lvKE&t=478s) know that security is not the first

[8:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=480s) thing that people hire for. Um but in

[8:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=482s) the age of AI, this is a very real

[8:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=483s) problem, and we need to make sure that

[8:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=485s) the startups we're working with actually

[8:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=486s) have somebody who can understand what's

[8:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=488s) going on from the security standpoint.

[8:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=491s) Uh so, some of the things we've seen go

[8:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=492s) wrong,

[8:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=493s) um

[8:14](https://www.youtube.com/watch?v=7A65O-0lvKE&t=494s) outright people just sending data to

[8:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=496s) their vendors, uh cloud servers, and not

[8:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=499s) following

[8:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=500s) any of what we've asked for. Um this has

[8:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=502s) been a problem in pilots. Uh thankfully,

[8:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=504s) all of our pilots run non-production

[8:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=506s) data.

[8:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=508s) Another one, like we kind of talked

[8:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=509s) about, um read write all default scopes.

[8:32](https://www.youtube.com/watch?v=7A65O-0lvKE&t=512s) So, there's a lot of really cool tools

[8:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=514s) out there, integrations, features.

[8:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=516s) They're really flashy. You can click a

[8:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=518s) button, and it'll integrate with

[8:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=520s) everything. And then you get a little

[8:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=521s) bit deeper and find out the only way

[8:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=523s) that it'll work is if you literally give

[8:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=525s) it read write all to everything, uh

[8:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=527s) which is a huge problem.

[8:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=529s) Another one, uh kind of along the same

[8:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=531s) lines, all or new beta features on by

[8:53](https://www.youtube.com/watch?v=7A65O-0lvKE&t=533s) default with each release. So, if you're

[8:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=535s) an enterprise, you don't want everything

[8:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=537s) just turned on with each release.

[9:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=540s) Um so, being able to control that,

[9:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=542s) and then

[9:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=543s) the line that we hear a lot, which is

[9:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=545s) we'll get you the security architecture

[9:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=547s) diagram next week.

[9:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=548s) Uh we do weekly check-in calls during a

[9:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=550s) pilot, and then we hear this over and

[9:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=551s) over. Uh it's not usually a great sign.

[9:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=557s) Uh the question that we often have our

[9:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=559s) CISO end up asking, which is um

[9:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=562s) what are you going to do if there's a

[9:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=563s) breach?

[9:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=564s) And we get this response, well, we

[9:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=565s) haven't had a breach yet.

[9:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=567s) Uh

[9:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=568s) with the subtext of we don't know what

[9:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=569s) we would do if we did.

[9:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=571s) Um okay, reliability, this is another

[9:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=573s) one. So, a control plane that actually

[9:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=575s) works.

[9:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=576s) We want to see every admin setting

[9:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=577s) available via API.

[9:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=579s) We want to see audit logs on config

[9:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=581s) changes. So, if there are five different

[9:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=583s) people who are given admin access and

[9:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=585s) somebody accidentally changes something

[9:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=587s) or does it because uh maybe it was

[9:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=589s) really late at night and maybe they had

[9:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=591s) too many drinks. Uh we actually want to

[9:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=592s) see what happened. Uh we want to be able

[9:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=594s) to control the rollout on these changes.

[9:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=597s) Uh we want to see real SLAs and a

[9:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=599s) reachable support engineer. That goes a

[10:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=600s) very, very long way.

[10:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=602s) So, uh things we've seen go wrong,

[10:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=605s) a lot of apps that are rapidly

[10:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=606s) prototyping, they're shipping so quickly

[10:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=608s) that they are maybe shipping updates

[10:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=610s) multiple times a day and there's a

[10:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=611s) really attractive little button that

[10:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=613s) says relaunch to update and it happens

[10:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=615s) across 3,000 people. We have no way of

[10:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=617s) tracking what's going wrong. Maybe then

[10:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=619s) like SSL certificates break in one of

[10:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=621s) the new releases and then we have no way

[10:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=622s) of tracking because everybody's on a

[10:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=624s) different version

[10:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=625s) um and we have no way of being able to

[10:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=626s) deploy at scale. Um that's really

[10:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=628s) challenging.

[10:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=630s) No documentation versioning.

[10:32](https://www.youtube.com/watch?v=7A65O-0lvKE&t=632s) So, support articles with new terms or

[10:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=634s) risks that are not actually in the legal

[10:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=636s) contract but show up in the website

[10:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=638s) somewhere in a random support page and

[10:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=639s) then we have no way of tracking what

[10:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=640s) they were before versus after and it

[10:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=642s) just says updated yesterday.

[10:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=644s) All of these are real examples, by the

[10:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=646s) way. I am not naming and shaming. Um I'm

[10:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=648s) just shaming. So,

[10:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=650s) uh maybe if any of you are familiar, you

[10:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=652s) can put it together. Um core API's down

[10:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=655s) for multiple hours during a busy trading

[10:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=657s) day. Uh that is a really big problem for

[10:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=659s) us because we run production systems. We

[11:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=662s) are trading billions of dollars.

[11:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=664s) Um this is a really big issue for us.

[11:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=667s) And then lastly, no SLA roadmap or

[11:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=669s) status page. Um the status page is a big

[11:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=671s) one.

[11:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=672s) Okay, last, legal issues. So, we don't

[11:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=675s) want anybody training on our data

[11:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=677s) regardless of what type of feature or

[11:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=678s) product it is.

[11:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=680s) Uh we also want to see a lot of

[11:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=681s) transparency in the sub processors.

[11:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=684s) Um any fourth-party risk becomes our

[11:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=686s) risk.

[11:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=688s) We want to see IP indemnification with

[11:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=690s) reasonable liability caps. Uh we do not

[11:32](https://www.youtube.com/watch?v=7A65O-0lvKE&t=692s) control the models, so if there's output

[11:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=694s) that is IP infringing, we don't want to

[11:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=695s) be held liable for it.

[11:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=697s) So, we have seen in pilots that people

[11:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=699s) claim they have ZDR, they have it

[11:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=701s) legally, but then they find out or we

[11:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=703s) find out later that they actually retain

[11:45](https://www.youtube.com/watch?v=7A65O-0lvKE&t=705s) some of our data because they say, "Hey,

[11:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=707s) we were looking at something and we

[11:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=708s) noticed this thing." And we're like,

[11:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=709s) "How did you notice that? You weren't

[11:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=710s) supposed to have this data." And they're

[11:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=712s) like, "Oh, yeah, you're right."

[11:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=714s) Um so, that's not great. If you say ZDR,

[11:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=717s) do ZDR. Um

[11:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=719s) next, every feature that is conveniently

[12:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=721s) beta with permissive data retention

[12:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=723s) clauses. So, we've seen some vendors who

[12:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=726s) they will stop releasing new features in

[12:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=729s) general availability. They will only

[12:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=731s) make them beta, and then the beta comes

[12:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=733s) with a secret little clause that says

[12:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=735s) that they're allowed to retain our data,

[12:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=736s) which is a very sneaky way of trying to

[12:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=738s) get our data. We don't like that. Um not

[12:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=740s) great.

[12:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=742s) Another one kind of similar is

[12:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=743s) fourth-party risk that's tucked away on

[12:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=745s) a random website page that's not listed

[12:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=748s) in the contract. Uh this is a really big

[12:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=750s) problem for us managing risk.

[12:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=754s) So, um

[12:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=755s) it was the best of times, it was the

[12:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=756s) worst of times. As a recap, the best

[12:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=758s) startups have security architecture that

[12:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=760s) actually works, support engineers who

[12:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=762s) respond, an admin API from the

[12:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=764s) beginning, a 90-day plan that deploys

[12:47](https://www.youtube.com/watch?v=7A65O-0lvKE&t=767s) into our infrastructure and cloud, and

[12:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=769s) success criteria that we write.

[12:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=771s) The worst AI startups don't have any

[12:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=772s) security architecture diagrams, no path

[12:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=774s) to a support engineer, no deployment

[12:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=776s) control or audit logs, no ETAs,

[12:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=779s) and salesmanship over solid product

[13:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=781s) building.

[13:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=782s) Um this is really just kind of a recap

[13:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=784s) of like what I have been through over

[13:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=787s) the last 2 years. Um I actually don't

[13:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=788s) think that any of this is novel,

[13:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=790s) um but it is codifying a lot of what I

[13:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=792s) feel like is good and best practice.

[13:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=795s) Um okay, so a new frontier model comes

[13:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=797s) out on average every 11 days, but your

[13:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=799s) architecture might be a decade or more

[13:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=801s) old. So, you've got a bunch of cool new

[13:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=803s) models, there's some amazing

[13:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=804s) capabilities is there, and then you have

[13:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=806s) profitability on the other side of it.

[13:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=808s) And what's in the middle? Maybe it's

[13:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=809s) your legacy architecture, probably a lot

[13:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=811s) of security and privacy issues, and a

[13:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=813s) lot of change management. Um ChatGPT has

[13:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=816s) only been out for 43 months. There are a

[13:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=818s) lot of companies who are still doing an

[13:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=819s) ERP migration that might have been from

[13:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=821s) 5 years ago. Um so, the timelines are

[13:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=823s) very asymmetric. Uh and I think that

[13:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=826s) sometimes we forget about that.

[13:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=829s) Um okay. So, my thesis again, half or

[13:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=832s) more of getting to AI native is unsexy

[13:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=834s) and has absolutely nothing to do with

[13:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=836s) AI.

[13:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=837s) Um AI models and products today can't

[13:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=839s) fix your legacy architecture. Although,

[14:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=841s) if any of you are startup people, that's

[14:02](https://www.youtube.com/watch?v=7A65O-0lvKE&t=842s) a great one to go for.

[14:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=844s) Um and it also can't run your change

[14:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=846s) management. These are unscientific

[14:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=848s) numbers that I'm putting up here, but I

[14:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=850s) hypothesize that 40% of getting to AI

[14:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=852s) native is AI models and products. The

[14:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=855s) other 60% is all the other stuff that no

[14:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=856s) one really likes talking about anymore,

[14:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=858s) uh which is like data hygiene, clean

[14:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=860s) architecture, having good integration,

[14:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=862s) strong enablement, and change

[14:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=863s) management.

[14:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=866s) Um I really look at AI as a flashlight,

[14:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=868s) not a band-aid. Um I really think that

[14:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=870s) AI shines a light on a lot of what's

[14:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=873s) already working or not working. It can

[14:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=874s) accelerate what's working really well,

[14:36](https://www.youtube.com/watch?v=7A65O-0lvKE&t=876s) and it breaks down very quickly when

[14:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=878s) things don't work well.

[14:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=879s) Um I don't think that it's a band-aid,

[14:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=881s) and I think that for everybody who's in

[14:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=882s) tech leadership, it's really important

[14:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=883s) to remember that if you have issues in

[14:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=886s) your technology estate, those need to be

[14:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=888s) addressed before trying to plug in AI

[14:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=891s) and just having everything rip. Um it's

[14:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=894s) it's not going to work.

[14:55](https://www.youtube.com/watch?v=7A65O-0lvKE&t=895s) Um so, hehehe again, maybe an unpopular

[14:58](https://www.youtube.com/watch?v=7A65O-0lvKE&t=898s) message, but I really believe that we

[15:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=900s) all need to start with the boring 60%. I

[15:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=903s) think that's where we all need to start

[15:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=904s) to get to the other side of the road.

[15:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=907s) So,

[15:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=908s) what did we learn as we shine the

[15:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=910s) flashlight internally?

[15:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=912s) Again, not revealing anything super

[15:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=913s) proprietary, but I do think these are

[15:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=915s) big picture lessons.

[15:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=916s) Number one, entitlements. Entitlements

[15:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=918s) need a new paradigm.

[15:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=920s) Uh there are a lot of people in a lot of

[15:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=922s) large enterprises who are over entitled,

[15:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=924s) under entitled. The entitlements model

[15:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=926s) and how it works and how it's managed,

[15:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=928s) all of that breaks down when you think

[15:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=930s) about agents and how quickly you want

[15:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=931s) agents to work and what you want them to

[15:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=933s) work on and their ability to exercise

[15:34](https://www.youtube.com/watch?v=7A65O-0lvKE&t=934s) judgment. Um the entire paradigm just

[15:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=937s) shifts.

[15:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=938s) Another one is cross-platform

[15:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=940s) integration moved up the stack. So, AI

[15:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=942s) is only as good as what it reaches and

[15:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=944s) we want it everywhere. Uh so, having

[15:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=946s) things that can integrate across

[15:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=948s) platforms is really important uh even

[15:50](https://www.youtube.com/watch?v=7A65O-0lvKE&t=950s) more than it already was.

[15:53](https://www.youtube.com/watch?v=7A65O-0lvKE&t=953s) Another one is centralized knowledge.

[15:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=954s) So, this is something that um Emil

[15:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=957s) brought up this morning in his keynote,

[15:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=959s) which is that basically we need thinner

[16:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=961s) agents and a smarter substrate. Um

[16:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=963s) centralized knowledge is really key to

[16:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=964s) that. So, all of your documentation, all

[16:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=966s) your support articles, everything that's

[16:07](https://www.youtube.com/watch?v=7A65O-0lvKE&t=967s) going on inside of your company that's

[16:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=969s) making it work, um all of that needs to

[16:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=971s) be centralized and easily consumable.

[16:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=973s) Even better if AI can help write that in

[16:16](https://www.youtube.com/watch?v=7A65O-0lvKE&t=976s) real time in a feedback loop. Uh that's

[16:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=978s) something we've been talking about with

[16:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=979s) some of our vendors.

[16:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=981s) Another one is a separate ecosystem for

[16:22](https://www.youtube.com/watch?v=7A65O-0lvKE&t=982s) experimentation. Um some companies may

[16:25](https://www.youtube.com/watch?v=7A65O-0lvKE&t=985s) need to get here.

[16:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=987s) That gap between your legacy

[16:28](https://www.youtube.com/watch?v=7A65O-0lvKE&t=988s) architecture and where you want to go

[16:29](https://www.youtube.com/watch?v=7A65O-0lvKE&t=989s) might be so vast that you actually just

[16:31](https://www.youtube.com/watch?v=7A65O-0lvKE&t=991s) decide, "Hey, maybe we need a separate

[16:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=993s) ecosystem to do a lot of this work,

[16:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=995s) figure out what does work and what

[16:37](https://www.youtube.com/watch?v=7A65O-0lvKE&t=997s) doesn't and then kind of go from there."

[16:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=999s) Um and that's something that we thought

[16:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1000s) about as well.

[16:42](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1002s) So, to just put a finer point on the

[16:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1004s) agents and the entitlement thing,

[16:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1006s) uh agents inherit your foundations. So,

[16:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1008s) I strongly recommend that everybody fix

[16:51](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1011s) their entitlements if they are not

[16:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1012s) working really well now

[16:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1014s) um because this is something that if you

[16:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1016s) think about the problems that you run

[16:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1017s) into when things go rogue, processes go

[16:59](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1019s) rogue, people go rogue, agents are going

[17:01](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1021s) to like 100X that problem. Um so, this

[17:04](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1024s) is really something that's worth

[17:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1025s) figuring out now.

[17:08](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1028s) So, to kind of recap uh as I wrap up

[17:11](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1031s) here, the recipe, if you are one of the

[17:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1033s) people in the first half who are raising

[17:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1035s) your hand on like, "What do I need to do

[17:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1037s) if I'm a startup and I want to work with

[17:18](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1038s) a really difficult large customer?

[17:20](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1040s) Millennium's got like 8,000 people. We

[17:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1043s) have very, very tight security,

[17:24](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1044s) compliance, regulatory requirements.

[17:27](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1047s) Um this is the stuff that we care about.

[17:30](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1050s) And we want to see more startups doing

[17:33](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1053s) work that allows us to work with them.

[17:35](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1055s) Um I really view this as like one of the

[17:38](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1058s) highest bars. We're probably not the

[17:39](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1059s) highest, um although we're probably

[17:41](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1061s) pretty close.

[17:43](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1063s) Um and I think if you can architect your

[17:44](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1064s) startup to work with companies like this

[17:46](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1066s) with this kind of architecture, um

[17:48](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1068s) you're probably going to be able to

[17:49](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1069s) satisfy basically everybody else.

[17:52](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1072s) Um on the other side

[17:54](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1074s) for anybody who's buying, uh these are

[17:56](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1076s) the things that I think again that kind

[17:57](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1077s) of that boring 60% that really deserves

[18:00](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1080s) a lot of work. Um off entitlements,

[18:03](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1083s) governance, audit logging, etc. Um these

[18:05](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1085s) are the things that I think we need to

[18:06](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1086s) have in terms of systems to get it

[18:09](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1089s) working on the other side of the

[18:10](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1090s) equation.

[18:12](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1092s) So,

[18:13](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1093s) that's it. Um

[18:15](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1095s) my only motivation here is to getting

[18:17](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1097s) stuff working better and having better

[18:19](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1099s) enterprise grade AI.

[18:21](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1101s) Uh that's a QR code to my LinkedIn and I

[18:23](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1103s) appreciate all of your time. Thank you.

[18:26](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1106s) >> [applause]

[18:40](https://www.youtube.com/watch?v=7A65O-0lvKE&t=1120s) [music]