AI Consulting in Practice — Nathaniel Whittemore (NLW), Superintelligent

Talk transcript

1. 0:00

   [on-hold music] Today, I'm excited to talk, uh, about something a little bit different.

2. 0:24

   So right now, uh, there's been-- the last couple of months have been an interesting time in AI. There's been a sort of surge in the air, uh, the narrative of an AI bubble, a lot of it driven by dubious studies, uh, like the MIT report.

3. 0:38

   And so what I wanted to do today is get into not so much the practice of consulting and transforming, but what organizations are actually finding value in right now.

4. 0:49

   So for those of you who don't know me, uh, there's kind of two contexts I bring to this conversation. The first is as the host of The AI Daily Brief, which is a daily, uh, news analysis podcast about AI.

5. 0:57

   The second is as the CEO of Superintelligent, which is an AI planning platform. So the different perspectives are sort of very high-level macro thinking about the news that's happening, and then a much more kind of ground-level view where we're spending a ton of time interviewing executives about what's going on inside their organizations.

6. 1:13

   And what we're going to talk about is sort of, one, kind of briefly in the first part, just the status of enterprise adoption, uh, as it currently stands. And two, um, and the more interesting part is we've been live with a study in, in the market for about a month now, collecting self-reported information about ROI around different

7. 1:31

   use cases. And this will be the first time, uh, this week was the first time I did some analysis on it. And so I'm going to share what people have, uh, what people have told us around the first kind of twenty-five hundred or so use cases that they've shared.

8. 1:42

   Um, so it should be pretty, pretty interesting stuff.

9. 1:45

   Talking about kind of enterprise AI adoption first, I'll go through this pretty quickly because it's, um, pretty well-known stuff. Uh, the short of it is enterprises are adopting AI, uh, in a, in a growing fashion.

10. 1:56

    Um, pretty much everyone is using it a-at least a little bit, uh, and increasingly they're using it a lot.

11. 2:02

    Uh, this year, I will need to tell none of you that there is a major inflection around, um, specifically adoption in the, uh, coding and software engineering, right? You saw a huge, huge uptick in this.

12. 2:12

    Um, there's a lot that's interesting about that from an enterprise perspective because it wasn't just with the software engineering organizations. Other parts of the organization are also now thinking about how they can communicate with code, build things with code.

13. 2:23

    Uh, but that's a huge, huge theme of this year.

14. 2:26

    Coming into twenty twenty-five, one of the big sort of thoughts that many people had was that this would be the year of agents inside the enterprise, right? That big chunks of work would get automated away.

15. 2:36

    And on the one hand, I think it's pretty clear that we didn't see some sort of mass shift towards automation, uh, writ large across different functions of the organization.

16. 2:46

    But when you dig into the numbers, there has been actually pretty significant, uh, shifts in the patterns of, of agent adoption. So this is from KPMG's Quarterly Pulse Survey, and it's a measure of how many enterprises that are a part of their survey, which is all companies over a billion dollars in revenue, have, uh, actual sort of

17. 3:04

    full production agents in deployment. So this isn't pilots. This isn't experiments. This is where they consider, uh, some agent that's actually doing, doing kind of work in a, in a full way.

18. 3:13

    And it's jumped from eleven percent in Q1 of this year to forty-two percent in their most recent study for Q3. So you actually are seeing pretty meaningful uptake of, of agents inside the enterprise.

19. 3:23

    In fact, I would argue based on our conversations that people have-- that it's moved more quickly through the pilot or experimental phase than people might have thought. Um, so much so that you're actually seeing now a big shift in the emphasis around kind of the human side of agents and how humans are going to interact with agents.

20. 3:42

    And it's involving a shift in upskilling and, and, uh, and enablement work. Um, you're seeing a decrease in the sort of resistance to agents as people start to actually dig in with them.

21. 3:52

    You're seeing more experiments like these sandboxes where people can interact with agents. So this is a big theme, even if it wasn't necessarily the dominant theme that some thought it might be coming into this year.

22. 4:02

    At the same time, it is absolutely the case that many, many, if not most enterprises are, broadly speaking, stuck inside sort of pilot and experimental phases. There is a, a lot of challenge around moving from some of those first exciting experiments to something that's more scaled.

23. 4:19

    Um, so this is from, uh, McKinsey State of AI study, which came out, I think, a couple weeks ago now. And you can see only seven percent of the organizations that they talked to claim or sort of see themselves as, as fully at scale with, with AI and agents.

24. 4:30

    And it's something like sixty-two percent are either still experimenting or piloting.

25. 4:35

    Interestingly, big organizations are on-- in general, a little bit ahead in terms of, uh, the organizations that are scaling as compared to smaller organizations. This has been a, a thing that we've noticed kind of throughout the trajectory of, uh, uh, of AI, um, adoption over the last couple of years, that you would think that perhaps smaller, more

26. 4:54

    nimble companies, uh, would be more kind of quick to adopt these things. But in fact, it's often been the opposite, with the biggest organizations making the biggest efforts. You can also see from the chart on the bottom that there's very sort of jagged patterns of adoption, right?

27. 5:08

    You're starting to see, uh, from, you know, last year if you looked, there's very similar kind of rates of experimentation across lots of different departments. You're starting to see some pretty big breakouts now, uh, with, for example, you know, IT operations kind of jumping out ahead of other functions.

28. 5:24

    I won't spend too much time on this sort of high performer piece, but I think the thing to note, because it comes back in, in, in some of the stuff that we found with our ROI study, is that you are also starting to see a pretty significant bifurcation between leaders and laggards when it comes to AI adoption.

29. 5:39

    And one of the things that tends to distinguish the companies that are leading is that they are just doing more of it, and they are thinking more comprehensively and systematically about AI and agent adoption.

30. 5:50

    So they're not just sort of doing spot experiments. They're thinking about their strategy as a whole. They're doing multiple things at once. And importantly, they're not just thinking about sort of the very kind of first-tier time savings or productivity types of use cases.

31. 6:03

    They're also thinking about how do we grow revenue, how do we create new capabilities, how do we create new product lines.

32. 6:10

    Overall, it's very clear that despite what is sort of, you know, the, the, the concerns in the media, that spend is going to do nothing but increase on this.

33. 6:19

    Um, so the bottom is the KPMG pulse survey again, and this is a, a, an estimation of the amount of money that these organizations intend to spend on AI over the next twelve months.

34. 6:28

    At the beginning of the year, it was a hundred and fourteen, which by the way was up from like eighty-eight in Q4 of last year. It's now up to, in their last study, a hundred and thirty million is what they expect to spend, uh, in the, in the year ahead, which obviously the, the total magnitude doesn't matter

35. 6:40

    as much as the change. Um, you also see the green charts are from Deloitte, and you can see ninety percent plus of organizations intend to increase their spend o-on AI in the next twelve months.

36. 6:51

    And as part of that, I think that you're going to see a much more determined conversation around impact and ROI, uh, which is a particularly thorny topic. But interestingly, there has been an increase in optimism over the course of this year around the realization of AI.

37. 7:09

    So this is from a different KPMG study, their annual CEO survey, which interviews tons and tons of CEOs. And if you look at the 2024 numbers, sixty-three percent of those polled thought that it would take between three and five years to realize ROI from their AI investments.

38. 7:25

    Twenty percent said one to three, and sixteen percent said more than five. This year, in that same survey, the number that said one to three years had gone up to sixty-seven percent.

39. 7:34

    There were now nineteen percent who said six months to one year, uh, and three to five years was down to just twelve percent. So huge, huge kind of pull forward of expectations of, of ROI realization.

40. 7:46

    The challenge is that ROI is really tough. So this is back to the pulse survey. Seventy-eight percent of those polled in that, in that survey said that they thought that ROI was going to basically become a bigger consideration in the year to come.

41. 8:00

    Uh, but also seventy-eight percent said that traditional impact metrics and measures were having a very hard time keeping up with the, with the new reality that we're living in.

42. 8:09

    And this is something that I've heard constantly over and over from CIOs and other people who are in charge of these investments, that the, the, the ways that we have measured impact of previous technologies and just previous initiatives are kind of falling flat with AI.

43. 8:22

    And so that got us thinking about the, the, the overall need that we have to just have more information. I'm not even talking about good, systematic information, just more information around what ROI looks like, what impact looks like.

44. 8:36

    And you know, I've got this great podcast audience. They're super engaged. And so we just decided, screw it, we're gonna ask them. We're just gonna ask them to report on what ROI they're finding from their use cases.

45. 8:47

    So this went up at the very end of October. Uh, like I said, as of this morning or when I looked-- last looked, we've had over a thousand submissions, uh, a thousand individual organizations rather submit something like thirty-five hundred use cases.

46. 8:58

    And, um, this is, uh, some, some of the first observations that we had around, um, kind of the first twenty-five hundred. So the impact categories, the way that we divided things was into sort of eight broad categories of impact, um, which will all, I think, be very intuitive to you guys.

47. 9:14

    Time savings, increased output, improvement in quality, new capabilities, improved decision-making, cost savings, increased revenue, and risk reduction. So basically, we're trying to think of like kind of a, a broad, simple heuristic for, uh, for, for kinda dividing or subdividing the different, the different ways that people are thinking about ROI.

48. 9:32

    And TLDR is that people are finding, uh, ROI right now. Um, now again, the caveats are that this is a highly enfranchised audience. They're listening to a daily AI podcast, and they are voluntarily sharing this.

49. 9:45

    So I think that, you know, there's, there's some caveating there. But you have forty-four point three percent saying that they're seeing modest ROI right now, and then you have another thirty-seven point six percent seeing high ROI.

50. 9:56

    For the purposes of a lot of these stats, high ROI will be significant plus transformational. Uh, only five percent or so are seeing negative ROI. And keep in mind, negative ROI doesn't mean that they think programs are failing.

51. 10:07

    That just means they haven't-- they've spent more than they've gained or, uh, uh, in terms of how their, their perception is. More than that, expectations are absolutely sky-high. Sixty-seven percent think over the next year they will see, uh, increase and high growth in their ROI.

52. 10:22

    So we've really optimistic sense from the ground view of where ROI is going to be in AI. Um, you even have the teams that are currently experiencing negative O-- uh, uh, ROI, fifty-three percent say that they're gonna see high growth.

53. 10:36

    So very, very optimistic. Um, as you might imagine, time savings is the default. It's the starting point for so many organizations. It represents about thirty-five percent of the use cases.

54. 10:47

    After that, increasing output, quality improvement, basically all those things that you would imagine around productivity are sort of like the dominant categories when it comes to these, uh, when it comes to these use cases.

55. 10:57

    When it comes to the specifics around time savings, you see a real cluster between one and ten hours, especially right around five hours. And I think this is interesting to call out because it's so obvious to all of us who are inside building these things, uh, whether you are a developer or an entrepreneur or just someone sort

56. 11:13

    of in and around it, how the, the vast breadth of opportunity that AI represents, new capabilities, things unimagined yet. It's hard to-- or it's easy to forget that if you save five hours a week or ten hours a week, you're talking about winning back seven to ten work weeks a year.

57. 11:29

    Uh, and that's very, very powerful. And when it comes to a lot of these enterprises, that is a very meaningful thing, even if it's not what they're ultimately in it for.

58. 11:39

    Interestingly, though, it's very clear that the story, although it might be, uh, have a concentration in time savings, is about much more than time savings. So this is the ROI distribution category, uh, uh, ROI distribution by organization size.

59. 11:52

    And this starts to get really interesting where you can see that there are some differences in where different size organizations are focused. So for example, the organization size between two hundred and a thousand people has a higher portion of their use cases concentrated in increasing output.

60. 12:09

    Now, we haven't taken the time yet to really figure out exactly what this means or even speculate on, on what this means. But I think it's interesting that this is a category of organization that has often reached a certain scale but is still very much striving for more and so seems to be focused more on use cases

61. 12:24

    that expand their capabilities. Same thing with, uh, when you start to divide things by role. You see a real kind of variance where, for example, C-suites and leaders, uh, are less focused on those time savings use cases and more focused on other things like increased output and, uh, and new capabilities.

62. 12:45

    In general, we're finding that C leaders, uh, and just sort of C-suite and, and leaders in general are even more optimistic and excited and, and seeing transformational impact than people who are in more junior positions.

63. 12:57

    Now, some of this might be sort of selection bias in terms of, um, what types of use cases you are focused on. If you are in that C-suite, you're thinking about things that inherently, if they work, are more transformational.

64. 13:08

    Uh, but it is notable that seventeen percent of, uh, of the use cases that, that people in those leadership positions have submitted, uh, they say have transformational impact and ROI already.

65. 13:19

    Uh, I'm gonna skip this 'cause there's-- we, we don't have time for too much. Um, you're seeing, uh, uh, interestingly, uh, a concentration, um, where the smallest organizations are getting more of that transformational benefit early.

66. 13:32

    Um, one of the things that I wanna do following this study is maybe do a sort of second round where we dig into what this one to fifty person, uh, size really looks like.

67. 13:44

    I actually think that whereas there might be a lot of similarity between a one thousand and a two thousand person organization, there could be a wild difference between a three-person, you know, small company and a forty-person company.

68. 13:56

    And so I'd really like to dig into that more. But you are definitely seeing a, a, a lot of impact in those sort of more small, nimble-moving organizations. Uh, as you might expect, coding and, uh, and software related or, uh, use cases have a higher ROI than average and a lower negative ROI than average.

69. 14:15

    Um, one really interesting kind of, you know, pulling on a specific category of use cases, risk reduction is our lowest category in terms of the percentage of use cases that, that that was their primary benefit.

70. 14:27

    So when you're filling out the survey, which is by the way at roisurvey.ai if you wanna check it out, uh, you basically only get to pick a primary ROI benefit.

71. 14:37

    We didn't want it to be super sort of, um, we wanted you to pick and, and hone in on the thing that was, uh, seemed most important or most significant.

72. 14:44

    And so only three point four percent have risk reduction as their primary benefit, uh, in terms of ROI categories. But it is by far, those use cases are by far the most likely to have transformational impact as, as the, a-as, as their outcome.

73. 15:01

    It's at twenty-five percent, so a full quarter of those, uh, have transformational ROI. And interestingly, I was having this conversation with a couple of my friends who work in sort of back office and compliance and risk functions, and this has been their experience as well, where there are a lot of, uh, a, a lot of the, the,

74. 15:17

    the challenges for those organizations involve sheer volume and quantity, uh, i-in ways that, that AI can be really helpful for. We also are finding some interesting patterns among organizations.

75. 15:29

    And again, this is where we get into some of the limits of this just being a whoever walks through the door of my listeners. We have a pretty heavy concentration among technology, as you might expect, industries and among professional services.

76. 15:40

    But we still have fairly decent sample sizes for some others. And in both healthcare and manufacturing, the use cases are meaningfully higher impact on average, uh, than the average across all organizations.

77. 15:51

    Um, which I think is, uh, it, it was kind of worthy of further study.

78. 15:56

    Last sort of part of this as I wrap up, you know, a lot of these use cases, as you saw, have to do with that sort of first tier that most enterprises are gonna be in.

79. 16:05

    Uh, increasing the amount of content that you output, increasing the quality of that content, just finding ways to win back, you know, your five hours a week. Um, but increasingly, there are automation and agentic use cases, and we are absolutely seeing that where those are the, the focus, where those use cases mention certain types of automation or

80. 16:24

    they mention agents, they wildly outperform in terms of the self-reported ROI from them. That's both on automation and it's on agents. And I think that that's sort of a, a trend towards where we're headed with sort of the next layer of more advanced use cases.

81. 16:40

    The last thing that, uh, from this sort of first, first look of observations is there is clearly benefits, and this goes back to, to what we saw with that McKinsey study as well, of thinking about AI and agentic transformation in systematic cross-organizational, cross-disciplinary types of terms.

82. 16:59

    Um, effectively, pretty much, uh, directly, the more use cases that a, a person or an organization submitted, the, the better they tended to see a ROI for. Now, there's lots of reasons for that, but I do think it speaks to that, that core idea that once you move beyond kind of your single spot experiments, there's a lot

83. 17:18

    of opportunity, uh, to, to sort of grow, grow the impact of the organization. So like I said, that is the, the first look. Uh, it's kind of the first two-thirds of these, uh, of these use cases.

84. 17:28

    We'll be open for another week, and then we'll have the full study out at the beginning of December. Um, I'm really excited, I think, heading into next year to see how we move from sort of generic conversations about impact, uh, and our gut senses about impact to a lot more random experiments like this to figure out where

85. 17:46

    the impact really is and, uh, and where we go next. So look at that. I'm gonna end twenty-seven seconds early and really throw off the time. But appreciate you guys all being here.

86. 17:56

    Uh, and again, if you wanna check this out, it's at roisurvey.ai. [upbeat music]