Understanding AI Stakes to Break Production Code

Talk transcript

1. 0:00

   [upbeat music] So the idea here is a lot of you are solving some of the most leading-edge AI, GenAI problems in the world in, in, in this room, and then a lot of you are trying ambitious things.

2. 0:23

   So main goal was to get folks, um, learning from each other. So think of this as a logical roundtable, uh, not, not very round physically. Um, so come away with insights about specifically your getting to production challenges by learning from each other.

3. 0:40

   So two rules of this AI roundtable. One is we're all-- we'll all be stronger if we figure this out together. Number two is Chatham House Rules. Um, that's for Peter's benefit.

4. 0:51

   Um, for us [REDACTED:origin] in the room, what happens in Nob Hill stays in Nob Hill.

5. 0:57

   Um, so l-let me actually kick it off by sharing some of my personal learnings. I'll go through maybe f- uh, five, five minutes or so of what I've learned in relation to getting to production, but specifically around the topic of stakes.

6. 1:11

   Um, and by stakes I mean, like, how high stakes is the problem that you're trying to solve. And I found there's a relationship between the bars and the obstacles that you run into as the stakes increase.

7. 1:21

   So I'll just share my own journey, and hopefully this is useful and food for thought. But as I'm going through it, think about what you're experiencing and the hurdles, 'cause what I'm gonna ask after I go through my own section is what, uh, what are some of the hurdles you are running into?

8. 1:38

   What are some of the ones that you haven't solved? What are some of the ones you have solved? So that we can learn from each other.

9. 1:45

   All right, so it all started like this. GenAI is a h-- GenAI is a hockey stick, right? Um, and then we kind of ended up with actual project experience where you can do, and I think Lucas said this earlier, lots of amazing prototypes.

10. 1:58

    It's so easy to build an amazing prototype, but then getting to production is harder. You have a higher bar. And so what a lot of folks have figured out is you can use RAG with vectors and all right, now it looks like a hockey stick.

11. 2:12

    And in some cases it does. In some cases you can get to production with this. But then there's still a class of use cases where that's not enough. And so what, you know, you'll hear from the Neo4j folks here, and if you go to the booth and attend Emil's talk and some of the other ones later, is

12. 2:28

    there's another unlock here with knowledge graphs and with GraphRAG. Um, again, like, will this be a hockey stick? I think the world moves in S curves, so this is probably more realistic and there's probably gonna be some other thing.

13. 2:40

    I don't know what that looks like. Um, but why this pattern? Um, so I-I've come to see a spectrum of application stakes, and it's a whole spectrum, right? But to kind of paint opposite ends and maybe something in the middle, if something is low stakes, then maybe it's more business peripheral.

14. 2:59

    Um, it's things like summarizing text. Here I have someone, you know, a children's bedtime story. Um, there's no single right answer. Uh, the stakes are a bit lower. Um, and then on the high end of things, like a wrong answer can have significant impact in terms of dollar value, brand or reputational impact, health and human safety, regulation,

15. 3:23

    fines, bias, all that stuff. And then, um, and then there's this in-between zone where, you know, it might be higher business stakes, but I've got some mitigations. Like, I've got a human in the loop.

16. 3:35

    Um, so, uh, so here you have a difference between at the far right, um, uh, pilot and in the middle, co-pilot. Uh, so there's kind of an interesting relationship between pilot, co-pilot, and, and stakes.

17. 3:51

    And, you know, the problem with high-stakes use cases is you bump into the meme, right? Which you've all seen. Um, and specifically it's things like needing enterprise domain knowledge, up-to-the-moment knowledge.

18. 4:02

    This is what RAG solves. Um, solving for h-hallucinations. Again, RAG solve this and depending on how you solve it, you're, you're loading the dice, loading the dice, loading the dice.

19. 4:13

    And then for-- depending on the stakes, your tolerance for error, you know, could be, you know, potentially gets very high, right? How t-- how tight should I tighten my bolts on my 737 MAX 9?

20. 4:24

    Like, let's not have a 0.001% chance of error, 'cause that's not good enough. Um, cool. So let's look at how we're representing data. So here's an apple, and think about how we conceptualize an apple as a human.

21. 4:39

    It's, you know, it's this multidimensional, multisensory kinda thing. Um, now let's look at a vector view of an apple. It looks something like this. [chuckles] Um, so I can find other fruit by doing a, you know, cosine similarity or spatial distance, um, operation with, uh,

22. 5:01

    w-with this representation of an apple. It's useful. Again, good for certain things, good-- not, not as good for other things. And if I characterize the behavior that you get with LLMs generally and with vector-based RAG, it's this, you know, kinda world of statistically computing, uh, the real world based on, um, word proximity statistics and word frequency statistics.

23. 5:25

    So you get this behavior that's kind of a little bit like the right brain. You know, it's creative, it's mostly right, sometimes really, really wrong. I don't really know why it's wrong, it just does its thing.

24. 5:35

    Uh, a bit of a black box. Enter knowledge graphs as a way to bump up the, the stakes on things. So this gives you a, um, more structured representation that is understandable by humans, can be reasoned upon by machines, and this gives you a bit more of left brain behavior.

25. 5:54

    So it can't do all the things that LLMs can by any means, but when you combine it with an LLM, you, you-- being able to bring in facts, reasoning, discernment, long-term memory, um, can create a better together.

26. 6:08

    So that's my quick- Um, sharing around stakes and unlocking things in different levels. So let me turn it over to folks in the room, and Peter will be coming around with a microphone.

27. 6:21

    So let me start with the question: Does anyone have... What's one learning that would've saved you a lot of time if you'd known it sooner?

28. 6:33

    Come on, all at once.

29. 6:34

    Anyone, just raise your hand.

30. 6:36

    Yeah.

31. 6:36

    Blurt it out if you're-

32. 6:37

    Well, maybe if, if no one's got one-

33. 6:39

    Yeah

34. 6:39

    ... maybe you have one, like, just off the cuff. It might inspire other people to have a story-

35. 6:45

    Um-

36. 6:45

    ... get the scope.

37. 6:46

    I mean, it, it comes back to stakes and, you know, un- understanding what those are upfront, um, for different applications. So w- we've heard a lot today about iteration and how important that is.

38. 6:58

    So definitely iterate. You know, don't overthink things initially. But, you know, to also think and, think and plan ahead, and that can, that can guide things like what tools you use.

39. 7:09

    Also locus of reasoning is, is something I think about, is if, if there's something-- if there's a question that has an exact answer and it's a der- deterministic question, then, you know, maybe you can use the LLM, which is inherently non-deterministic, to complement some other technology that can give you a deterministic answer, which you have.

40. 7:28

    You've got databases, um, and, and other kinds of reasoning engines. And then use the LLM to do what's, it's exceptionally good at, which is maybe take the question, uh, turn it into, um, a query.

41. 7:41

    Take that query, formulate it back into some text, and then maybe modulate that text depending on who the audience is. If it's going to regulator, then very serious. No jokes.

42. 7:50

    If it's going to, you know, a, an outbound marketing email, then it can, uh, maybe go to, uh, be phrased differently depending on who the target audience is. Is it someone in high school?

43. 8:01

    Is it a retiree? Um, yes, I see a hand, though-

44. 8:04

    Oh

45. 8:04

    ... though over there.

46. 8:04

    Cool. Let's go. [laughs] Here we go.

47. 8:07

    Going around.

48. 8:08

    Ah, so I'll go here.

49. 8:10

    Um, I've, uh... One, one learning that I wish I had had before is instead of asking the LLM to solve a problem, right, like paste in Excel and say which one's the outlier, just ask it to write the code to solve that problem, which it can easily do, and then you can leave the LLM out of the

50. 8:27

    loop. Um, right. You say, "Find the outlier." It will do some Python outlier machine learning thing, and then you have the answer.

51. 8:35

    Cool. Yeah. So the LLM might not be good at certain kinds of reasoning, but it can write code that will do the reasoning, which is a pretty amazing hack, actually.

52. 8:43

    Thanks. Um, I think you were-

53. 8:45

    Oh, everyone's getting involved now

54. 8:46

    ... I know. I'll, you, you, you work out who's next.

55. 8:48

    I'll... Wait, I'm sorry. Who had their hands up? Oh, let's go here.

56. 8:52

    Yeah. So one of the things we learned was, um, when we were kind of getting excited about the RAG and indexing and doing the semantic search, and when we put it to production, that our customers were not used to doing queries like this.

57. 9:07

    They always resorted back to keywords. It's almost like they were-

58. 9:11

    Mm-hmm

59. 9:12

    ... like-

60. 9:12

    Like search

61. 9:12

    ... trained to do this.

62. 9:14

    Yeah.

63. 9:14

    And so getting them along, uh, and so we had to write, rewrite, like is this question actually a query for keywords or is it like the, like a query they ask like a, a question?

64. 9:26

    So that was a learning that, you know, we, we made, uh, while putting it into production. And, uh, we put a lot of effort in making the search well, [laughs] but we, we, we didn't ask upfront like, well, is this actually how you wanna use it?

65. 9:39

    And so that was one learning there.

66. 9:41

    And so did you have a sort of curriculum to help people understand how to ask questions and that they can do more than just keywords?

67. 9:49

    Uh, so yeah, we, we didn't put attention to this. We were just-

68. 9:52

    Yeah

69. 9:52

    ... assuming like, yeah, everybody's now gonna use like-

70. 9:54

    Yeah. Yeah

71. 9:54

    ... putting questions in. So that was, you know, the surprise and the learning there, like-

72. 9:58

    Yeah, that's cool. Yeah

73. 9:58

    ... we needed to ease them in.

74. 9:59

    You need to understand what the technology's capable of, and they can do more than just-

75. 10:02

    Yeah. But it's like the-

76. 10:03

    ... what you're used to. Like it's not a Google search box

77. 10:03

    ... adapt the user to how the new thing is supposed to work, uh-

78. 10:07

    Yeah

79. 10:07

    ... in a way. Like that was a learning.

80. 10:09

    Cool.

81. 10:10

    Thanks.

82. 10:13

    Um, I think the lesson is kill more projects than experimentation. So right now, I think one of the big lessons we learned is we did a lot more experimentation, and r- if I have to look back, I will focus on few that will go all the way to production.

83. 10:29

    Yeah.

84. 10:30

    Because a lot of these experimentation, even with the best effort, will never meet the user expectation on accuracy. So if we can kill and... a lot of projects, but focus on one or two that can, we can take all the way to production is probably where I will focus a lot more.

85. 10:46

    And people spend too much time, uh, on not being focused.

86. 10:50

    And when, when deciding what project to focus on, I imagine there's an element of how achievable is it and how valuable is it. H-how, how do you-

87. 10:57

    The thing is for us, like it, it... Can we get to the level of accuracy that the user is expecting?

88. 11:03

    Yeah.

89. 11:03

    Commercialization is expensive, but we can do it to 90, 95. That's not good enough. Why are we spending time on it now? But work on the use cases that will go to them, where 90% is good enough.

90. 11:15

    Yeah. So matching up with user expectation of accuracy, and then those projects are bound to succeed and go with those. Cool. Thanks.

91. 11:23

    Okay.

92. 11:23

    My learning is, uh, expect less and use more common sense.

93. 11:28

    Yes.

94. 11:28

    It's a funny one.

95. 11:29

    Yeah. I like that.

96. 11:37

    Thank you.

97. 11:38

    One learning that I've experienced from our customers when they're building deployments is, you know, it's really easy to walk in and say, "I want the best model, and I want it to run as fast as possible."

98. 11:50

    But when you're making, you know, actual cost to quality trade-offs and cost to speed trade-offs, you know, you can, you can really overspend versus what you really need if you don't have a great understanding of what you're actually trying to build, how fast it actually needs to run, that kind of stuff.

99. 12:08

    Yeah. Yeah. That in, in a lab with a small number of questions, you're naturally gonna go for ... the best accuracy, but then enter production if it, you know, if, if, if you're burning a year's budget within a day, which is very possible with this stuff, then maybe you need smaller models or multiple...

100. 12:24

     Yeah. Gr- great one.

101. 12:26

     Actually, I just had one pop up in my own mind, um, that I remember seeing, which was when Bloomberg were, uh, training, I think it was Llama 2 at the time-

102. 12:36

     Mm

103. 12:36

     ... to handle a lot of the financial data that they had, and it passed really well on all the evals and everything. And then [laughs] they found out when Llama 3 came out, it actually outperformed their fine-tuned model they'd spent all this time producing.

104. 12:47

     Um, and so I'm not entirely sure what the takeaway from that is, but it's almost this idea that if you just wait for the models to get better, then you kind of save work.

105. 12:56

     But then I guess that's not necessarily gonna fly in the enterprise where [laughs] it's like, "We want something working today."

106. 13:00

     Well, th- this is a great... I'd love to hear what people in the room are experiencing, 'cause there's this real tension of, all right, if I just wait a couple years, like, this stuff is all gonna figure itself out and it'll be way better, but who wants to wait a couple year?

107. 13:12

     Like, we, we wanna get out and get competitive advantage. [laughs]

108. 13:16

     Not a couple years, right? Like, we're seeing crazy advances.

109. 13:18

     Yeah, yeah. You wait, you wait a few [laughs] weeks.

110. 13:20

     So like, we're, we're just focused on, like, the full path and getting it end to end, and then, like, in parallel tracks, optimizing every part of that, that journey, right?

111. 13:29

     Yeah.

112. 13:29

     So trying to end the end shipping customer value, and then, oh, our data engineering part of that was like a clutch and that's not gonna scale, so let's get people working on that.

113. 13:37

     And it's like an interface between every step of this chain that we're putting together. And so, like, if I encounter an issue where my LLMs aren't performing and it's like, "Oh, no, we gotta use, like, 3.5 Sonnet and it's gonna cost us a lot," well, just wait, you know?

114. 13:50

     It's gonna come down a little bit.

115. 13:52

     Yeah.

116. 13:52

     And then kind of just keep optimizing as these improvements happen across that stack.

117. 13:56

     Yeah, yeah. The, the, the bars I was showing earlier are naturally all rising, like, across all those dimensions, and if you understand what the pipeline looks like, then you can just drop stuff in, right?

118. 14:08

     Figure out your pipeline first-

119. 14:09

     Yeah

120. 14:09

     ... and then you can just iterate on each one of those components.

121. 14:12

     Yeah.

122. 14:12

     That's, that's what's been working really well for us.

123. 14:14

     Cool. I like... So o- o- o- one question to drill into that is, I can see pipelines getting more complex with more agentic stuff and so on. Like, how do you rec- like, that one is a moving target too.

124. 14:26

     How do you see handling-

125. 14:27

     It's, it's-

126. 14:27

     How are you handling that?

127. 14:28

     It's not something we've figured out yet. Um, obviously looking for tool, better tools to help orchestrate that.

128. 14:34

     Yeah.

129. 14:34

     We know we are making improvements on various parts of this thing, but, like, I've had to slow down the team. Like, we're making too many changes at once now across the different things.

130. 14:42

     Mm-hmm.

131. 14:43

     So it sounds like we have to slow down a little bit with interfaces and contracts between the layers so that we can iterate on one side and not affect the downstream on another.

132. 14:50

     So I don't know that yet.

133. 14:52

     Cool. Great. Anyone else on this question?

134. 14:54

     Just, just to piggyback.

135. 14:55

     Yes.

136. 14:55

     Um, when you're thinking of the latency improvements you can get if you have a solution that might take you months to implement, you have to evaluate that with the trade-off and say, "By the time we finish this, will a new model have come out?"

137. 15:08

     Yes.

138. 15:08

     So I think the time it takes to implement new solutions, there should be, like, a max, like a ceiling. Like, if it takes more than however many months, you're just wasting your time because it's gonna be throwaway.

139. 15:18

     I love that. Yeah. If it's a two-year project, then you're-

140. 15:21

     Yeah

141. 15:21

     ... w- why bother? Like anything-

142. 15:22

     Basically-

143. 15:23

     ... you do now is gonna be irrelevant.

144. 15:24

     Right. I need another solution.

145. 15:25

     Yeah.

146. 15:25

     Basically.

147. 15:26

     Yeah. Love it.

148. 15:30

     I'm gonna dash over here.

149. 15:31

     I could also just shout.

150. 15:32

     Yeah. It sometimes works. But, um- [laughs]

151. 15:35

     Thank you. Yeah, one, I've-- all of that I really resonate with, and one thing that we've seen is companies that have succeeded have avoided premature optimization. So obviously you want to be sort of aware of latency in production and, you know, make something that'll be realistic.

152. 15:50

     But the teams that have sort of started with that and gone, "We're gonna use a small fine-tune model," often incorrectly conclude that a task isn't possible. And also there...

153. 15:59

     people are often able to take data from a larger model and fine-tune it later. And so I always think of, like, moving down the model size stack or fine-tuning as optimizations, and you want to validate the product need and whether it works, whether it's possible, usually with a better model first.

154. 16:14

     At least we've seen that be much more successful.

155. 16:16

     That's a great... I, I've, I've heard that too, like start with GPT-4.0 or I guess the, now, now maybe Claude 3.5 and, um, if that can't do it, then the smaller models probably won't.

156. 16:28

     And, and I guess you have an exception for data that's highly proprietary, and that's, that's where fine-tuning and RAG come in.

157. 16:34

     Yeah. We do the same thing. Just optimize the best out of the box, and then we kind of slowly bring it down-

158. 16:40

     Yeah

159. 16:40

     ... to more performant, cheaper models, hopefully trying to get the same accuracy out.

160. 16:46

     Cool.

161. 16:46

     I think building up on, on something that you mentioned as well earlier when, for example, you're using a, a, a large language model to turn unstructured data to structured data to then feed it into, like, traditional machine learning models.

162. 16:56

     Mm-hmm.

163. 16:57

     One of the key learnings for us was if we're, um, if we're building our eval pipeline from the get-go, which, uh, is really starting to be echoed, you know, trying to figure out what your eval should be, start, get it, get it ready from the start, is to understand do you have the right data to build your

164. 17:12

     eval pipeline? And if not, start from day zero to make sure that you generate that data for your eval pipeline. Because if you leave it to stay too long and then realize, oh, we've, we've built an LLM model and it's in our app, but we cannot evaluate whether, uh, the fine-tuning on the prompts makes it better or

165. 17:31

     worse, you're then spending a lot of sort of long leg time on, on figuring out what your eval pipeline should be. So our key learning was really start, as, as you start building your app, also start with your eval pipeline.

166. 17:43

     Make sure that you have the data that's there.

167. 17:45

     Cool. That, that's maybe a corollary of the, okay, you, you need your DevOps kind of framework, but then you need your eval framework, and then your data framework is closely related to that, is making sure you have the right data.

168. 17:59

     Yeah, and the quality's there.

169. 18:06

     Um, one of the things we learned is not hiding AI from the end user. So we're- Like explicitly, for example, generating the responses slowly, so as a hint for the user-

170. 18:19

     Hmm

171. 18:19

     ... that they, they should be more vili- vigilant. We're using specific colors in the UI to make it clear, uh, the icons that they click on. So kind of make sure that they understand that there's a special thing they need to do.

172. 18:32

     And then when they wanna check, we give them as much tools for them to validate the answer that was generated to see what, what went wrong, what were the sources, what kind of, uh, went in there.

173. 18:43

     So that's one of the lessons that, uh, not to abstract this away and kind of hope for the best. But, uh-

174. 18:48

     This is related to your moderate expectations point.

175. 18:50

     Yeah.

176. 18:50

     If the user can actually see the thing coming out, then that's a trigger that, um, all right, it's not instant, it's coming from some machine, I need to consider-

177. 18:59

     Yeah

178. 18:59

     ... things. Cool. I have, um, I wanna ask if anyone,

179. 19:05

     um, dealing with regulation or something where explainability is a big issue, and if so, how are you solving that?

180. 19:17

     What sort of industry are we talking? Like, it, like a medical something or-

181. 19:19

     Um, yeah, I, I could think about medical, like if you're trying to pre- if, if the end game is prescribe something or provide some treatment.

182. 19:26

     Yeah.

183. 19:26

     I can think about, um, equipment maintenance is another one.

184. 19:29

     We work in the behavioral health space, um, and a lot of their... There's a lot of regula- regulation, and the one way we've determined to solve it as a kind of a core principle is just always human in the loop right now.

185. 19:42

     Yeah.

186. 19:42

     Like, we're only in the copilot stage. In fact, I tell people we'll never get to the pilot stage, but that's just to not scare them. But, uh-

187. 19:49

     Yeah

188. 19:49

     ... we're, we're, we're very focused on copilot right now, uh, and making sure there's a human in the loop. Um, but as part of that we have to give transparency to the inputs that generated the outputs, so they can kinda see things side by side.

189. 20:01

     Um, if we make recommendations or we pick some data out that we think is interesting, uh, for the particular case, uh, that they're working with, um, they can, they can trace it back to the, the root data.

190. 20:13

     And so do you find the inputs are sufficient for getting the humans in the loop to trust the answer enough? 'Cause I, I... So you've got inputs. Another way to do it is here are the references that, you know-

191. 20:27

     Yeah

192. 20:27

     ... but then they're not gonna read the three 50-page papers maybe.

193. 20:30

     We, we, we... So if it's, if it's doing a, a lookup, a knowledge lookup, we're showing the source documentation that we find based on their query, uh, so they can see it kinda side by side-

194. 20:41

     Yeah

195. 20:41

     ... um, with those citations. Um, if it's doing a generation, uh, so let's say we're summarizing data that was collected in a therapy session, um, we show the data side by side with the actual summarization of that.

196. 20:51

     Uh, it's kinda like on a paragraph by paragraph basis so that the users have complete control of the, the resultant summary. They can change it, edit it. It's theirs at that point.

197. 21:02

     And then are you... Last question. Are you using the, um, judgment of the human in the loop as like [REDACTED:username] to improve the producer?

198. 21:12

     Uh, yeah, we run our own eval on any of our output with a separate, um, stage just to make sure that, like, did the input... Does the, does the output match kinda the inputs?

199. 21:21

     And that's just a quick one. It's also for a, a safety perspective. But then we use the, whether or not the user edited or, and/or just gave us thumbs up, thumbs down.

200. 21:29

     Yeah.

201. 21:30

     Uh, and then we do our own clinical evaluation at the end, uh, on our own, with our own team, just making sure, like we're just... Everywhere we need to, we're putting in that eval stage to make sure that we, we know what, uh...

202. 21:43

     we're comfortable with what's coming out.

203. 21:45

     Cool.

204. 21:46

     Uh, I'm just gonna go over here. Just one thing affects, uh, whether we can do one more or multiple more is, is our next speaker here? 'Cause I've not-

205. 21:54

     Yes.

206. 21:55

     Oh, fantastic. Cool. [laughs] Okay. Right, unfortunately this will have to be the last one then for now, but, uh, I assume this, this discussion-

207. 22:01

     Yeah

208. 22:01

     ... can continue at lunchtime as well, 'cause it seems everyone, there's so many shared concerns, um, and stories that people have got in the room, so it would be a great thing for people to discuss at lunch.

209. 22:11

     Yeah.

210. 22:11

     Absolutely. Here we go.

211. 22:13

     I'll keep this short. So a hu- human in the loop is clearly a solution, but that's not always possible. So what we do is give options. So this is, uh, uh, in a, one of our use cases is personal loan recommendation.

212. 22:27

     Hmm.

213. 22:27

     It's regulated.

214. 22:28

     Yeah.

215. 22:28

     And then so we give, rather than, um, ever say, "This is the best loan for you," we give three options and give explanation for these, and so it becomes less deterministic and less, uh, and, and that, that's how we take care of the regulation.

216. 22:42

     That helps.

217. 22:42

     Okay, cool. Yeah. So g- give a few options, and then do you do that, do that directly to the end user or to the human in the loop?

218. 22:47

     It's end user because we cannot have human in the loop.

219. 22:49

     Right. Right.

220. 22:49

     But I think the-

221. 22:50

     Yeah

222. 22:50

     ... the three options that we se- select are clearly t- show some more logic to it than just coming, that's coming out of the LLM.

223. 22:58

     Cool. Thanks. Well, let's definitely keep this one alive throughout the day as you're-

224. 23:03

     Thank you

225. 23:03

     ... mingling with each other and lunch and so on. But, uh, thanks. I appreciate the participation. [upbeat music]