How To Build an AI Strategy That Fails

Talk transcript

1. 0:00

   [upbeat music] All right, everyone.

2. 0:18

   Welcome. Hamel and I are absolutely thrilled to be here with you to basically teach you how to build the definitive guide to completely, utterly, and spectacularly messing up your AI strategy.

3. 0:31

   I actually couldn't have really wished for a better foil, Grace's lead in, than this because we're not just talking about minor setbacks here. We're going to take you through a way to create full-blown, company-crippling, career-ending failure. [laughs]

4. 0:50

   Grace talked about best practices, but we're here to embrace worst practices. In fact, we're going to make sure you know how to completely torpedo your AI projects and ensure you alienate everyone that you work with. [laughs]

5. 1:06

   How does that sound?

6. 1:07

   It sounds great to me. Um, so before we begin, it... We might as well start with some introductions. Um, we have no agenda here. It's just a sequence of steps.

7. 1:17

   But I'm Greg. I'm an executive leader, and I've spent years in the C-suite crafting AI strategies. I'm now a co-founder of an AI startup. But previously, I was the chief product officer at Pluralsight, an executive leader at other companies, um, and I've had a front row seat to how executive teams can transform clear strategic opportunities into labyrinthine

8. 1:40

   disasters.

9. 1:42

   And I'm Hamel. I'm a machine learning engineer and independent consultant who has worked with many companies on AI. I've witnessed every conceivable way AI strategies can fail, and I found it really fascinating how creative people can get with their failures.

10. 1:57

    That's right. So you could say that together we're kind of like the dream team of disaster. We've advised or maybe we've, you know, interacted with representatives from numerous companies.

11. 2:08

    We even, even have this fancy website. Um, but for today's presentation, we've decided to live and breathe the great words of the late Charlie Munger, who said, "Invert. Always invert."

12. 2:19

    So let's get started. All right. The first step to failure is to make sure that you begin to divide and conquer your own company. This is key if you're destined to fail.

13. 2:34

    You got to embrace the disconnect be- between willingness to pay, price, and cost, the keys to creating value, by contemplating unreasonable goals. And you should, especially everyone here in the audience, know by now you, you have to make sure you go and attend every AI industry conference, right?

14. 2:53

    But never go back and talk about what you learned with your team. The point, just like Moses here parting the Red Sea, is to create impenetrable silos and incentivize secrecy between your teams.

15. 3:06

    So let's get into it. [clears throat] So I talked about the value stick and willingness to pay in the, the prior slide. But here it's really important for us to adhere to the anti-value stick.

16. 3:20

    You got to embrace it because it's the opposite of everything good and useful when it comes to value creation and being strategic. And today, that's our guiding principle. You might be thinking that WTP means willingness to pay, but here it's wishful thinking promises.

17. 3:35

    You got to tell your customers that AI is going to do absolutely everything for them. Your new systems are going to write their emails, walk their dog, solve climate change, and achieve world peace.

18. 3:46

    But don't really worry about the details. Just promise the moon. You know about price, right? Well, for us, that's another acronym. Particularly ridiculous infrastructure costs everywhere. [clears throat]

19. 4:01

    I mean, that was a mouthful, sorry. Uh, buy the most expensive GPUs. Don't bother with any cost benefit analysis. Just max out the company credit card. Think of this as an investment in something.

20. 4:13

    And cost? Well, it's that cascade of spectacular technical debt you're about to run headlong into. Um, you need to think about building systems so convoluted, so intertwined, that even you as an executive can barely understand it.

21. 4:28

    You know about job security, right? This is a key to guaranteeing it. Think about it this. When it inevitably breaks, no one's there except for you. And finally, if you know about value, you know about WTS or willingness to sell.

22. 4:44

    For us, it's why this system? Well, the answer, and I mean always, is because AI. There's never any further explanation needed. No board is ever going to question you.

23. 4:55

    It's like magic, but much more expensive and less reliable.

24. 5:00

    So step two here is when you start to define your strategy, right? Here's the first key. Fake any diagnosis you might be thinking of. Grab last year's annual report or, or operating plan and just start highlighting random paragraphs, preferably the ones you understand the least, and declare, "AI must fix this."

25. 5:20

    Don't bother talking to anyone who actually does the work.

26. 5:24

    And your guiding policy should be both incredibly ambiguous and vague. Something like become the global AI leader in everything. [laughs] Except don't define what everything means. That's someone else's problem.

27. 5:39

    Totally. And your action plan, simple. You need an AI-powered SEO tool that guarantees top Google search results, even if you sell garden gnomes, right? And a generative art plugin that creates NFTs of your CEO's cat.

28. 5:51

    And of course, an AI drone lunch delivery service because synergy? Uh, announce all this at your next company all-hands meeting, and you get bonus points if you wear a shiny suit and use the word disruptive at least a dozen times.

29. 6:05

    And the last point on this slide is about timelines. But timelines are for companies that intend to finish projects. What we recommend is you embrace perpetual beta. Just create a massive backlog in GitHub and stick all those highlighted financial reports into that Greg was mentioning earlier.

30. 6:22

    Great strategy.

31. 6:24

    But you know what strategy really works? Just create a four thousand-page document that you post in all your Slack channels and just erode people's willpower to engage with the material. [laughing]

32. 6:36

    And with these tidal wave of documents, um, you know, in words, Greg, isn't there a strategy that you have about jargon?

33. 6:45

    There certainly is. The point is to communicate in such a way that nobody understands. Drown everyone in a tsunami of jargon. Say things like, "Our multimodal agentic transformer-based system leverages few-shot learning and chain-of-thought reasoning to optimize the synergistic potential of our dynamic hyper- hyperparameter space."

34. 7:03

    If you say it with confidence, you probably will have absolutely no idea what you just said.

35. 7:09

    Remember, the goal is to look incredibly smart, even if nobody understands a word you're saying. The key is obfuscation.

36. 7:18

    Yeah. You might be tempted to do something like defining a very cogent, clear business-on-a-page approach, like in the, in, in Advantage, but never be too tempted.

37. 7:31

    One of the most effective ways to cause dysfunction in your organization is to use jargon everywhere, and use jargon strategically to hide the jobs to be done. For example, I had a mental health client.

38. 7:46

    Instead of saying, "We need to write a prompt," we would just say, "Hey, we're building agents." And what that did is it made sure that the mental health, health experts

39. 7:57

    were not in the room and didn't know how to participate. And that's exactly the result you want.

40. 8:03

    That's right. Just like Hamel reduced those mental health experts' mental health, I like to do that as well. So instead of saying, "Hey, let's make sure the AI has the right context," I just talk about RAGs.

41. 8:16

    And don't say, "Make sure users can trick the AI into doing something bad," just say, "Prompt injections."

42. 8:22

    Yeah. And the key here is to encourage engineers, not the people who might best understand your customers, to write prompts, because what could possibly go wrong?

43. 8:33

    Look, we know that translating everyday English language into jargon can be really difficult, so we made this guide for you. And this guide will help you divide your organizations, just like Greg was talking about earlier.

44. 8:47

    Just like Moses.

45. 8:48

    The link is right here. But remember, making everything, even writing prompts, seem super technical and out of reach for everyone is what you wanna go for. [laughing]

46. 8:59

    All right. Just a brief recap. We talked about how to s- seed your, your division, how to start to define your strategy, how to communicate it. Now we're onto mobilization, right?

47. 9:09

    Because you gotta do something with that giant backlog. Well, some of you might know about Geoffrey Moore, but I've never heard of him. Today, we're pioneering a very new revolutionary framework, which is about zoning to lose.

48. 9:22

    It's designed specifically for failure.

49. 9:26

    Just randomly assign AI tasks to people with absolutely no relevant experience. For example, outsource your data review to offshore Q&A teams who have very little context about your business.

50. 9:38

    Yeah. And most importantly, you might be tempted to use the incubation zone to bootstrap new AI ideas, but the goal is to launch from here completely untested, bug-ridden AI chatbots directly to your customers.

51. 9:52

    As Hamel mentioned, never worry about beta testing. Dis- disregard quality assurance. Just ship it straight to production, because what's the worst that could happen outside of a potentially career-ending PR disaster?

52. 10:06

    So if you do it sort of right, it should feel something like South Park. Um, you're gonna yank all your best engineers from potentially supporting your revenue-producing products, wait a while, and then profit?

53. 10:20

    No. Actually, it's gonna feel more like total collapse. And because you're so disorganized, we can now transition to...

54. 10:28

    Look, at this point, your organization is in complete disarray, but it's time to do the deed and burn it all to the ground.

55. 10:37

    So the most effective way to start doing this is to focus on tools, not processes. And those problems that you created earlier, and other ones that may exist, don't analyze them.

56. 10:48

    Don't try to understand them. Just throw tools at them. [chuckles] So if your RAG system isn't retrieving the right documents, just buy a new, more expensive vector database.

57. 10:58

    Yeah. And if you need to measure progress, just use every off-the-shelf evaluation metric you can possibly find. Never bother customizing them to your business needs. Just blindly trust the numbers, even if they make no sense.

58. 11:13

    Oh, and, like, you know, we've been talking a lot about agents today. If they're not working, just pick a new framework and vendor. Fine-tune without any measurement or evaluation.

59. 11:22

    Just assume it's gonna be better, because it's kinda like alchemy with a lot more electricity.

60. 11:29

    Exactly. You don't need to look at our design metrics. Evals, that's a vendor problem. Just plug in a tool, and it will solve all your problems. Greg, I really love how you demonstrated exactly what we're going for here with Whac-A-Mole.

61. 11:46

    Every time you see a problem, hammer it with a tool. If another problem comes up, hammer that with a tool. If the same problem comes again, hammer it with a different tool.

62. 11:55

    You get the point.

63. 11:56

    Yeah. Hamel, I really appreciate being meme fodder to help you get your point across.

64. 12:02

    Look, I wanna emphasize, you should adopt a mindset that evals are a vendor problem.

65. 12:10

    Just realize that there should be a one-size-fits-all solution. Let the vendors figure it out. You're too busy being an executive. And if you really wanna do this properly, you need to create a dashboard that looks like this, with every off-the-shelf metric that you can gather.

66. 12:32

    The more metrics, the better. It doesn't matter if the metrics track with outcomes or real failure modes. Make sure the numbers are unintelligible so you don't know the difference between a 3.5 and a 4.5.

67. 12:46

    Keep hoarding random metrics until you find one that's going up and to the right. Then you can claim success. [laughs]

68. 12:54

    And look, maybe you might have a hard time figuring out where you can come up with these generic metrics, but we got you.

69. 13:02

    Just adopt the ones from eval frameworks. In fact, adopt all of them. Let your eval metrics guide you blindly, and never ask whether they actually measure success. Again, the more numbers you have, the better.

70. 13:16

    Yeah. I personally like to optimize for cosine similarity, BLEU, and ROUGE, ignoring actual user experience. And I said it once, and I'm gonna say it again, never cross-check with domain experts or your users.

71. 13:29

    Because if an LLM says it's accurate, who are we to argue? We are their humble servants, after all.

72. 13:35

    Amen. Now it's time to unveil the most potent technique we have in our toolbox, and it's avoid looking at data. Seriously, just avoid it. Keep a blindfold next to you at all times.

73. 13:49

    You happen to bump into data by accident, put that blindfold on.

74. 13:53

    Yeah. Data? That sounds really messy. Let's let a tool handle it because you can absolutely 100% trust the AI's output without ever looking at it yourself.

75. 14:03

    Looking at data, that's an engineering problem. You're a leader. You have more important strategic things to do, like having meetings about meetings. [laughs]

76. 14:16

    Besides, developers, they always have more domain expertise than your business teams.

77. 14:22

    Yeah. And we know that ultimately, by this point, your customers are really your best Q&A. And hopefully, you have lots of them, and they'll complain if something is wrong, maybe eventually.

78. 14:33

    But more importantly, you gotta trust your gut. It got you this far in life, right? And feelings are always a reliable substitute for data, especially when you're making million-dollar decisions.

79. 14:46

    If you have trouble trusting your gut, just put the blindfold on. It'll get you right back in touch with those feelings.

80. 14:53

    So now we know by, we know by now that engineers are all coding wizards, and they're gonna handle [clears throat] everything. It doesn't really matter if they haven't spoken to a customer in years because you can quickly forget about the fact that there might be simpler options, like using spreadsheets to annotate and look at data.

81. 15:10

    Say after me, "Remember, this is beyond me."

82. 15:15

    Great advice. And just, it's not enough for you not to look at the data. You have to make sure no one else is looking at data. [laughs] And the best way to do that is to put your data in complex systems that only engineers can access, and it's not available to domain experts.

83. 15:32

    Right. So, like, instead of using a simple spreadsheet or perhaps an Airtable, like up on the screen, as an executive, you should insist on buying a custom data analysis platform that requires perhaps a team of PhDs to operate and understand.

84. 15:47

    Remember those bonus points? You get more of them if it takes six months to load this thing and errors incessantly.

85. 15:55

    So there you have it, the ultimate foolproof guide in under 20 minutes to achieving total AI failure.

86. 16:03

    If you follow this advice that we've given you here meticulously, it's guaranteed that you're gonna waste time, resources, and alienate all the people you work with. And as far as I'm concerned, that's the ultimate success that you can have.

87. 16:17

    Sure is. So for more advice, it's actually real, do visit AI-Execs. Um, we also have an O'Reilly book, the same material, coming out February 27th. Uh, and so while this talk was inverted, you know, our lived experience really isn't, and we're always very eager to help you on your journey.

88. 16:36

    So find us after this presentation at the Q&A speaker booth. Thank you so much. [outro music]