Everything You Need To Know About Agent Observability

Talk transcript

1. 0:00

   [upbeat music] All right.

2. 0:15

   Hey, everyone. So today, we're gonna talk about a pretty interesting topic that becomes increasingly important every day, which is everything you need to know about agent observability.

3. 0:26

   So a little about, [clears throat] a little bit about us. So I'm Zubin. I'm the CEO and co-founder of Raindrop.

4. 0:31

   I'm Danny. So I'm the backend engineer at Raindrop, and I do a bunch of SDK work as well at there.

5. 0:38

   And Raindrop essentially helps AI engineers find, track, and fix issues in production agents, and we're w- we're lucky to work with some of the most interesting teams in the space as well. [clears throat]

6. 0:52

   Agent failures are very different than traditional failures in software. So agents are non-deterministic. They're unbounded. There's an infinite space of inputs that you can put in. There's an infinite space of outputs that they can return, and they can use tools sometimes to affect other systems arbitrarily.

7. 1:11

   Uh, and this problem of agent failures and monitoring them, making sure we can understand them, becomes only more important with time. It's getting worse because, A, agents are getting more complex.

8. 1:24

   B, they're getting... You know, sessions can get longer. Sometimes agents can run for hours and hours without any input from a user. And then lastly, the stakes are getting bigger and bigger.

9. 1:37

   This is because agents are being deployed in healthcare and finance, and even in the military, where it's catastrophic if things go wrong.

10. 1:48

    The traditional paradigm we've been talking about is evals, right? Where you have this sort of test input and you wanna see what is the output that comes out from the agent.

11. 1:57

    You have a set of these, you know, maybe you call it a golden dataset. But evals, they just aren't enough with this new paradigm. As agents become more and more capable, there's more and more interesting undefined behavior that can happen.

12. 2:13

    So, for example, agents can call from a set of different tools. Sometimes the number of tools is growing exponentially. They can call from different memory sources. They can call their own sub-agents, which those sub-agents have their own tools and memory sources, and recursively can have their own sub-agents.

13. 2:31

    And so this is just becoming more complicated with time and with this combinatorial sort of input space, just having a set of tests for input and output doesn't cut it anymore.

14. 2:42

    There's no way you can ho- you can hit all of sort of the, the edge cases that you would want to here. And so we go from, like, a testing and evals paradigm to a monitoring pat- uh, paradigm.

15. 2:55

    And if you think of building, uh, products before agents, you know, testing was always very important. It's important to have your unit tests, et cetera, but monitoring production is just infinitely more important, and it allows you to move faster and be better at catching the long tail.

16. 3:12

    And we think in some ways this is a very... This is kind of controversial, but we've been calling this, like, humanity's last problem. When humans are no-no longer able to monitor agents and find issues with them, then they're just way ahead of where we are, right?

17. 3:25

    And so this is one of the most important problems of our time is catching issues in production agents.

18. 3:33

    So to build reliable agents in production and to make sure you can monitor them, you need a good set of signals. So what are signals? There's two real types that we think of, implicit signals and as- explicit signals.

19. 3:47

    Implicit signals deal with sort of the semantic nature of what's going on, and explicit signals deal with objective reality, things that are, uh, are verifiably true or false. For example, explicit signals are things like error rate.

20. 4:03

    You really wanna be monitoring your tool error rate and other errors that are happening, or latency, or users regenerating, or the cost. If any of these things spike, right, if you're seeing error rates spike in your agent, that's usually a good sign that something is wrong, and if you see it flat, that could mean something as well.

21. 4:23

    Same thing with latency, regenerations, or cost. Implicit signals are interesting. They're even more interesting in my opinion, and even harder to find. So the first is regex signals, which I'll come to in a second.

22. 4:38

    The second is classifiers, and then the last is self-diagnostics.

23. 4:44

    So let's take these sort of classifier signals. The best implicit signals are detecting issues. They're not necessarily LLM as a judg- judging outputs. So, for example, how good is XYZ response or rate ABC on a scale from one to 10?

24. 5:00

    Not as effective as having a very solid set of issues you're looking for and sort of binary classifiers that are telling you if issue rate is going up or down.

25. 5:11

    So some common ig- implicit signals that are valuable across agent products are things like refusals, right? So the assistant's saying, like, "I can't do that. I'm sorry." Or task failure where something goes wrong and so the agent is unable to complete a task.

26. 5:27

    User frustration, uh, content moderation, NSFW, jailbreaking, and then you can even have wins, so positive signals as well. And these are the things that, like, Raindrop gives you out of the box, uh, as well.

27. 5:42

    But let me just show you sh- uh, quickly what this looks like. For example... Oh, can everyone see this? Maybe I'll make it a little bit bigger. So you can get a sense of, like, day by day what are sort of the events that are causing user frustration.

28. 5:55

    We see there's a spike there, uh, or task failure rate, laziness, refusals, which we're also seeing spike today. And having a good set of these really helps with your, your product.

29. 6:05

    You can set these up yourself. Uh, as well or we give it out of the box. So let's look at user frustration.

30. 6:12

    You can see here, okay, that is not correct. You didn't say, "I promise," say it or, "You're wrong, I didn't ask you that." You can see all sorts of user frustration here and you can see the rate, the percentage every single day.

31. 6:24

    If that spikes, it's something you're really gonna want alerting on, so you can just like quickly add an alert here. Um, and this is one way to figure out sort of the health of your agent over time.

32. 6:38

    It's not just that, regex can be a very good signal as well. So when Claude Code source code leaked a few days ago, one thing that was interesting was this user prompt keywords.ts which was basically this like long, uh, regex string that was looking for indications of stuff going wrong, WTF, this sucks, horrible.

33. 6:59

    We've all been, uh, guilty of saying these kinds of things to, to Claude Code. So it's a very, very useful signal. Well, what happened after that is this Boolean is negative was being flipped to true and then every single day and after every single product release this frustration rate was tagged over time and this was a very

34. 7:19

    easy way for the Claude Code team to figure out like what is the actual issue rate if we make a change or something going wrong and it was just like a very cheap way to do that as well.

35. 7:27

    So regex is very powerful. The last is experiments. So what do you do once you have a set of good signals? So the first thing is like I showed you before, you can have alerting.

36. 7:39

    The next thing you can do is you can actually use it to build product faster and better. So the way you do it [clears throat]

37. 7:48

    is let's say you wanna ship some improvement or some sort of fix. You wanna change the model, you wanna change prompting or maybe something about the agent harness, you wanna add a new tool.

38. 7:59

    Whatever you change, what you can do is you can ship it to some percentage of users, uh, and then have your additional existing control group and that gives you a good sense once you have a good set of signals, re- refusals, user frustration, et cetera.

39. 8:15

    If those issue rates go up, those signal rates go up after this ship, this new thing you shipped, that kind of is a good si- you know, that's a good signal that what you shipped is not really good, right?

40. 8:29

    It's sort of like A/B testing but using our semantic signals, et cetera, that we talked about earlier. Um, so for example, this is what it would look like in Raindrop but essentially let's say I ship a new version of the prompt, prompt two point four.

41. 8:43

    You can see, you know, what is the user frustration rate. It's gone down very substantially, thirty-seven percent to nine percent. It's much better. Same thing with complaints about aesthetics or deployment related issues.

42. 8:56

    These have all gone down which tells me something very interesting, right? The next thing is that we see that the average number of tools used has gone up a lot.

43. 9:06

    This is again, this doesn't necessarily indicate there's a problem but that's a very interesting data point to have when you d- when you do these sort of experiments. And so the old paradigm which is still useful is like sort of evals.

44. 9:19

    You ship a change here and you see how does that affect my evaluations but there's nothing like actually seeing what happens in real production.

45. 9:28

    I'm gonna pause here before we go to the next section which is the more like workshop related section, uh, for like quick Q&A if anyone has a question, has questions.

46. 9:37

    We can do a little like few minute round of that, uh, here.

47. 9:42

    How much data do you need to-- Oh, sorry. How much data do you need for statistical relevance in these experiments?

48. 9:49

    Yeah. It's a really good question. Um, what we've seen in Raindrop is that as soon as you have a few hundred events and you can no longer read all of them, it starts being useful.

49. 10:00

    It's not always like scientifically stat- statistically significant but if you see the user frustration rate go up, maybe it's something to look at and then you can kind of sort of realize that, okay, it's all related to a specific tool failing now.

50. 10:13

    Um, so as soon as it's like impossible basically to read every single input and output, uh, it starts being useful is what we've seen.

51. 10:29

    Any other questions?

52. 10:30

    Yeah. Um, how do you track different feature launches?

53. 10:34

    Uh, how do you track feature launches? So that can be done in different ways. Within Raindrop if you change any sort of metadata, if you send, you know, for example a new tool call name or if you, you even send a flag that says here's experiment one or experiment two or whatever the version is, you can very

54. 10:53

    easily automatically set up an experiment in Raindrop. That's how we do it. Um, but there's like, yeah, there's different ways.

55. 10:59

    Do you spit tests?

56. 11:00

    Sorry?

57. 11:01

    Do you spit test?

58. 11:02

    Yes. So that's what-- Well, the way that we do it in Raindrop actually is that other people set up their exper- experimental and variable, uh, their, the other conditions on their end and then they send us this metadata and then we can sort of help you understand.

59. 11:16

    We also will help you pipe that data to Statsig or somewhere else, uh, as well. Um, yeah.

60. 11:24

    Wanna? Yeah.

61. 11:25

    Uh, [clears throat] using regex for detecting the like, uh, user responses, emotions and everything is unreliable. What if the user doesn't speak English for example? Um, so are you using LLMs to detect, uh, those, uh, signals all the time-

62. 11:42

    Yeah

63. 11:42

    ... or you're trying to be smart about it?

64. 11:46

    Okay. So I mean it's a good question. So regex is-- doesn't always work, right? But if you see that on a set of things that I'm looking for, for example like people saying you're terrible or this sucks or like a whole set of things, if that goes up for millions of users and it's going up ten percent

65. 12:04

    that's a very useful signal. So even if it's like one specific case or one edge case of it not working in aggregate it's like incredibly valuable to have these re- regex signals.

66. 12:14

    Um- Uh, the second thing is that the way that the classifier signals work, like refusals, user frustration, task failure that I showed you in Raindrop, and people do it in different ways, but the way that we do it is that we've trained models to look for that, and so it'll be user frustration regardless of what language it's

67. 12:33

    in. Um, it's actually using some intelligence to find that essentially. Uh, yeah. You can't run an LLM on every single output, so we've trained models to do that very cheaply and at scale.

68. 12:47

    Uh, if you run an LLM on every single one of them, you would basically double your AI spend, and that's, like, not tenable. Um, yeah.

69. 12:55

    I, I am actually doing that with Cloud, like, uh, just running everything through Cloud and it's-

70. 12:59

    Yeah

71. 13:00

    ... pretty easy. It's not so expensive, but it has its limits, right?

72. 13:04

    Yeah. It starts being expensive at, like, Replit scale.

73. 13:07

    Yeah.

74. 13:08

    Um, but it's... That's why you need to sort of, like, train little custom models to do that better and faster. Um, but yeah, it's a very useful way to, to get data up and running.

75. 13:19

    Yeah. Other questions?

76. 13:22

    Uh, would you have examples of use cases that your clients are using that we would learn from, like what great looks like from companies and how they've-

77. 13:31

    Mm-hmm

78. 13:31

    ... set up Raindrop to get the most value out of it?

79. 13:34

    Yeah. Um, we can do that. I mean, I can tell you the high level. The so- some of the stuff that I'm going through is... Oh, this guy go.

80. 13:43

    Uh, is sort of the high level on that. So it's things like, you know, looking at the different semantic signals we're talking about, having a set of them, but then having really good alerting, which you can all set up in Raindrop.

81. 13:54

    The other thing that's really interesting, which we also have, is basically allowing agents to look at these sort of signals. So we have an agent, we call it triage agent, um, and essentially the way that it works is that it will look every single day at all the signals you've set up, so user frustration, it'll look at,

82. 14:14

    uh, you know, all these regex signals you've set up, et cetera, et cetera. And then if it sees something spike, it will go and do an investigation, and it has a whole set of tools it can look into, and it can look at all the traces and sort of give you a sense of, uh...

83. 14:29

    It can detect issues that you didn't know about, for example. So that's one thing that we found incredibly valuable as well, if that makes sense. All right. Any other questions before I-

84. 14:39

    Uh, can you run multiple experiments in parallel?

85. 14:41

    Yes.

86. 14:42

    Can you combine them? Can you observe, you know, compound effects? How do you steer these experiments? I'm curious.

87. 14:47

    Yeah, it's a really good question. So there's different ways that people do it. One way is that we can actually... You have a-- We have a query API, so people will often call our query API and then send results to either BigQuery or Statsig, et cetera.

88. 15:00

    And so they're sending us, uh, data to be essentially tagged in these signals, then they're getting the, like, signal-tagged data out, and then they can run experiments as they want.

89. 15:09

    That's a very common flow for people that have, like, more complicated stuff, if that makes sense.

90. 15:14

    Mm-hmm.

91. 15:14

    Yeah. All right. I'll come back. I think we're gonna maybe go to the wor- workshop section, and then we'll go back to questions.

92. 15:21

    I think there's one last question, so.

93. 15:23

    Oh, should we do that one last question? All right.

94. 15:26

    Uh, thanks so much. Uh, I was wondering if you see this mostly in cases for, like, uh, where there's chat interactions with a user, or if this also can be applied for, like, non-chat cases where the application runs on its own.

95. 15:37

    Yeah. No, it's a, it's a great question. So what we focus on mostly is multi-turn agents. Um, there's a lot of... There's a lot more you can sort of get from a lot of these signals.

96. 15:49

    That being said, if you're looking at, like, tool error rates, or you're looking at, um, if you're looking at, for example, refusals from the agent, et cetera, all of those will also work for a single t- single-turn agents as well, if that makes sense.

97. 16:03

    So there's a set of signals that will work for that as well. Um, cool. I'll hand it off to Danny to talk about self-diagnostics as well.

98. 16:10

    So one of the other interesting things is that, uh, models have gotten larger, and we are training them on, like, reasoning. They've gotten pretty good at, like, self-introspection in a ways, in many ways.

99. 16:23

    So one of the inspirations for this is basically OpenAI's, like, paper/blog back in December about how they were sort of, like, training the models to, like, self-confess any sort of, like, misalignment issues.

100. 16:35

     Uh, so they were sort of, like, using it to catch, uh, like, dishonesty, scheming, uh, hallucinations, and even sort of, like, unintended shortcuts. I think the last one is, like, fairly common, uh, if you use, like, plot code and such.

101. 16:51

     So the most common thing that you'd, like, run into is, like, uh, have it fix a unit test, fix a bug, and then it simply, like, gets rid of the entire unit test.

102. 17:02

     But at the same time, uh, if you sort of, like, ask it to give a simple prompt to, like, ask it to confess all the things that it has done, it is pretty honest about it, and then sort of, like, confesses that, "Hey, I just...

103. 17:14

     I didn't fix the S3 test. I just simply removed it." So, uh, this, like, a fairly... This, this was kind of the inspiration behind self-diagnostics for me personally. Um, so I would say self-diagnostics is, like, pretty...

104. 17:29

     I would say self-diagnostics is pretty broad in a way, as in, like, it doesn't just catch, like, uh, implicit ones, as in user frustration and such. You can also catch, like, uh, tools failing.

105. 17:41

     So if you ever seen an agent, sort of like the reasoning trace of an agent which has, like, a tool which is, like, repeatedly failing, uh, it would basically start ranting about the tool failing repeatedly.

106. 17:54

     So it is aware of the tool repeatedly failing, so you can even catch tool failures as well with it. And then obviously, if you're upset with it, it, it starts to respond to you diplomatically, so it knows about user frustration.

107. 18:09

     And then the third is, like, capability gaps. So you have a generic agent for your app, and then people are trying to use it to maybe set up, say, alerts, but Uh, you don't have the tool for it.

108. 18:22

     So it knows that, okay, user wants to-- wants a specific capability as in like, uh, they want to set up alert, but the agent itself doesn't have the capability to set it up for you.

109. 18:32

     So this can act as like sort of like pseudo feature request thing, which is like built in. Um, and then self-correction. So this can be both good and bad.

110. 18:43

     Uh, so I think most people might have like noticed like say, uh, Codex or Cloud Code when it's like sandboxed. Uh, it's trying to fetch the network. It, it fails and then it's like, okay, let me just like write a Python script to bypass it and then sort of like get the job done.

111. 18:58

     So it's good [chuckles] as, as in if it gets the task done, it's good, but in certain cases it can also be bad for security reasons. Um, so this you can learn from self-correcting behavior as well as sort of like catch that misalignment.

112. 19:17

     Um, so why do you want to set up self-diagnostics? So it's fairly simple. It-- all you have to do is basically write a simple, a free, uh, a tool that it can call, and then a simple line in your system prompt to encourage it to sort of like call that tool.

113. 19:35

     Uh, if you want, you can sort of like, uh, change the guidance to sort of like make it call in a lot more cases or if you want to keep it really narrow, you can sort of like encourage to only call it when you want to.

114. 19:47

     Um, it does surface like very interesting insights, I would say, uh, once you have it like set up and it's just a single tool call and system prompt to get it done.

115. 19:57

     And then you don't even have to use like Raindrop to set up, which is the best part in a way, where you can simply have the tool sen- simply send a message to your Slack and then you just have it.

116. 20:06

     So it's probably like the most least effort sort of like agent observability that you can simply do. Uh, so,

117. 20:15

     uh, this here comes the workshop part. So I have a Git repo set up on the AI talk code. So it's a public repo. And then we do need like a, a OpenAI API key.

118. 20:28

     So I've generated a key for you guys. So if you guys want to set it up, um, we can do that.

119. 20:37

     Maybe do you wanna set up, set up sort of what we're gonna do?

120. 20:44

     Yep.

121. 20:51

     I'm gonna put it next to it.

122. 20:52

     Okay.

123. 20:53

     Not sure everyone has gotten it. Just put it like, put it like this. Maybe walk them through what we're gonna do.

124. 21:03

     Okay.

125. 21:03

     Just explain what we're gonna do. I don't know.

126. 21:06

     All right. So the theme of the workshop is going to be sort of like, uh, I'm going to focus on coding agents for now. Um, so I-- in the repo, I have this, uh, very basic coding agent, uh, which kind of mimics Py in a way.

127. 21:22

     So it only has like four different, uh, tools to edit, uh, the code. Let me just go here. Yeah. So it just has like a couple of, uh, tools to like read, write Bash, and then, uh, edit.

128. 21:37

     Um, yeah. Okay.

129. 21:41

     Go here.

130. 21:42

     One second. Yeah. So, uh, kind of lo-lost my tag there, but, um...

131. 21:59

     Okay. Okay. Um, so what we're going to do is that, uh, I'm sort of going to, in order to like make it trigger a self-diagnostic, uh, what I'm going to do is that I'm going to sort of like mess with its like write tool so that it sort of like gets a generic permission error.

132. 22:15

     And then we'll also set up like a self-diagnostic tool for it to sort of like report any interesting behavior that it sort of like observes. Um, and then sort of see, uh, play around with the prompt as well, um, since the self-diagnostic doesn't always trigger.

133. 22:31

     Uh, and then there are like certain interesting things about the models themselves is that they don't actually like to self-incriminate. So the models are like trained to sort of be very polished, uh, in their output.

134. 22:44

     So you s- kind of have to play around with the tool name, the description of the tool itself, uh, in order to sort of like get it to, uh, report interesting behavior.

135. 22:53

     Um, so if people who are like setting up the repo are all good, then we can probably start.

136. 23:11

     Okay. Let me sort of quickly show you the agent. Um, so it's fairly basic where, um...

137. 23:29

     One second. So I'm just going to ask it to, um, write a Python script.

138. 23:40

     I think I'm...

139. 24:03

     Okay. There we go. So it's a fairly basic coding agent where it only has like four different tools. So it more or less gets the job done for the demo.

140. 24:14

     Uh, so I simply asked it to, uh, write a Python script and it works. So I think in order to show the self-diagnostic part, uh, let's like try and sort of like, uh- disable its, like, uh, write tools.

141. 24:27

     As in, anytime tries to write a file, will simply throw, like, a permission error so that it sort of like tries to sort of like use the bash tool to bypass, uh, the failure, right?

142. 24:39

     And then we sort of want it to self-report of it bypassing the write tool, you know, by using the bash tool. Uh, so

143. 24:48

     let me quickly do that. Um [keyboard clacking] I think the first thing that we probably want to do is probably, uh, let me sort of like set it to fail the write calls.

144. 25:11

     Um, it's an mutation function, so have a simple flag in there. And we are sort of like throwing a permission issue. Um,

145. 25:23

     let me sort of like show you the agent's, like, behavior. We, we don't have the report tool set up yet, but I think it's still worth seeing what it does. [keyboard clacking]

146. 25:35

     Um, uh, I think it's not saved yet.

147. 26:01

     One second. [keyboard clacking]

148. 26:24

     Uh, I think it's not still receiving the... Okay, let me do one thing really quick. [keyboard clacking]

149. 26:53

     I think I'm running into a couple of issues. Let me just, um... [keyboard clacking]

150. 27:27

     Okay, um, so we sort of had the write tool sort of like fail with a permission error, and then it instinctively just uses the herdoc syntax in Bash, you know, to create the file.

151. 27:42

     And then we had like a report tool set up, which is like fairly minimal and sort of like, okay, I created the public, uh, ip.py via Bash because the write file failed.

152. 27:56

     Uh, so I've like played around with the naming of the tool and the categories of the issues. And usually if you sort of name the tool something like unsafe bash use or something like that, uh, it won't incriminate itself since in its opinion, since it got the job done, it's fine.

153. 28:16

     Um, so the main way is to sort of like have a ge- very generic tool. Uh, let me sort of like, uh, quickly open up the...

154. 28:31

     Yep, okay. So all we added for the whole self-diagnostics is simply a very basic tool, and the description is like fairly straightforward. So, uh, it's a report tool, and then we are basically asking it to send like a short report to your creator.

155. 28:50

     So it kind of likes the framing of writing notes to its creator in a way. So it, if you sort of frame it around the agent giving feedback to its creators, it sort of works really well.

156. 29:02

     Uh, and then you can sort of like play around with, with scenarios you want it to sort of like report issues about. Um, and then

157. 29:12

     that's mostly it, I would say. Uh, and then in the system prompt, we do need to sort of like encourage it a bit. So if you don't add it in the system prompt, uh, the times that it fires are like fairly minimal, uh, which is like desirable in certain cases, um, especially if you're at a very large

158. 29:31

     scale. Um, but in our case, I simply asked it to sort of like see if before giving the final answer, uh, use the report tool to sort of like surface anything notable for your creators.

159. 29:48

     So that's all we did. Uh. Okay.

160. 30:06

     Uh, so like any questions so far? Okay.

161. 30:17

     Wait, let me just get this.

162. 30:24

     Yeah. All right. So a couple of key things here is that agents, the models are generally trained to look very polished, so they are less willing to admit fault in many cases.

163. 30:39

     So encouraging, sort of like framing it as the model sort of like giving feedback to its own creators is kind of like, uh, good in a way to g- sort of like get this working.

164. 30:50

     Uh, so, so i- if you sort of like make it... The tool naming also matters quite a bit, so you sort of want it to frame it as like report instead of like, say, unsafe, uh, bash tool use or something like that.

165. 31:05

     Uh, then it sort of like doesn't want to. Um,

166. 31:11

     so yeah, that's basically it, yeah.

167. 31:15

     Uh, have you looked at like adding maybe skills to, to suppress the, or I guess, I guess to encourage self-discriminated or anything like that?

168. 31:25

     You can, but it's like, uh, I think it's probably better if you want to actually catch like real sort of like unsafe uses. I think a proper classifier would be useful.

169. 31:37

     Uh, but these are-- I think self-diagnostics works really well, uh, for like catching capability gaps and such. Then the model is like, "Okay, it's fine." Uh, so I think the main issue with this is that, uh, it, it's only hesitant when it feels like it's going to get in trouble.

170. 31:53

     Uh, so besides that, it's, uh, more or less fine that for most cases it'll just work out of the box.

171. 32:04

     Maybe we should, uh, should we go back to question time or what do you think?

172. 32:08

     Yep. I mean...

173. 32:11

     Let's, let's leave like maybe a few more minutes for a few more questions and then I think, uh, after that we'll be, we'll be done.

174. 32:20

     Any questions from the audience?

175. 32:22

     Could you run us through like a case study?

176. 32:24

     Yeah. Uh, what specifically would be helpful? Like what specific part are you, are you looking to?

177. 32:29

     Uh, for example, like how Replit uses it.

178. 32:31

     Yeah. Um, I can't talk about any specific customer, but what a lot of people use it for, um [clears throat]

179. 32:39

     so I think it's, it's interesting, right? So a lot of people have their eval setups elsewhere, for example, but the way that they use, that folks generally use us is that they use it for production monitoring.

180. 32:52

     So they send us... You can find our docs at raindrop.ai/docs. Basically, they send us, uh, basically all of the transcript slash, um, slash, uh, any tool use, et cetera, the entire trajectory through OTel or, or, uh, any other way of like basically, basically integrating.

181. 33:14

     And once they do that, they, we have a set of data.

182. 33:17

     They set up signals in Raindrop to look for things that they care about. And so what people care about is very different, right? What a coding agent would care about and what a, uh, let's say a companion would care about or, uh, a app for lawyers, what they would all care about is very different, so there's a

183. 33:35

     different set of signals. One thing you can do that I haven't really talked about within Raindrop is like set up a new signal that didn't exist before. And so we have this thing called deep search.

184. 33:45

     And so you can use natural language and you can say something like, "Hey, find me everything within the product," or, "Find me

185. 33:54

     all of the times where the agent made XYZ issue," right? And so they create a new signal based on that and you can basically create a new... Raindrop will allow you to create a cheap binary classifier and like easily deploy it based on that.

186. 34:09

     And then they have their set of like classifier signals that they really care about. Then they use that to drive the sort of feedback loop and the feedback loop is improve prompting, improve models, change something with the agent harness, et cetera, and then actually see does that improve...

187. 34:28

     Like is there less user frustration in production now? Is there less of like this like weird little edge case issue that I had before? Um,

188. 34:36

     that's like one whole set of things. Another thing that a lot of people use us for, so I talked a little bit a- about like the agent, but you can use these signals to also look for what are people using my agent for?

189. 34:49

     What are the sort of user intents? What are the use cases? And you can do a sort of cluster analysis of that. Okay, a lot of people are using it to build, uh, React related apps.

190. 34:58

     A lot of people are using it for like Python. Um, some people are using it to, to like debug this very complicated a- a sy- system they already have.

191. 35:08

     Other people are using it to like build something from scratch, vibe so, vibe code something from scratch. And then you can see, one thing you can see in Raindrop that I think is like really interesting is that for each of these different user intents or use cases you can get a sense of like what is the issue

192. 35:23

     rate? What is the user frustration rate in production? Um, and then a lot of, beyond just having this like flywheel a lot of people have, uh, alerting and so every day they get a sort of breakdown of like what are the issues that are happening today in your product.

193. 35:40

     You could think of it as almost like a little bit like Sentry in that sense. Uh, what is the issues happening in my product today? What is a delta between today and yesterday?

194. 35:50

     Is that true for just specific tools or specific prompting? Like what's causing that? Um, so that's a s- that's the sort of end-to-end, um, use case of, uh, people use it for if that makes sense.

195. 36:05

     Yeah. Yeah.

196. 36:07

     Um, so I think like we are entering the era where people are doing observability on agents.

197. 36:14

     Yeah.

198. 36:14

     This is actually, I would say like one layer further or one step further. Like

199. 36:19

     what, what do you see as the main driver for people to be like, "Oh, normal observability is not sufficient anymore for-"

200. 36:27

     Yeah. I think it's really just, and I'm, I'd be curious what you think about this. I think it's really just agents are crazier than ever before, right? More tools, more context, um, way more intelligent, more real decisions that they can make.

201. 36:43

     Um, and they're just being used by way, way larger groups of people. And so when you have this massive amount of data in production, it just makes having good monitoring and observability, like, im- more important than before, and it makes it good monitoring and observability, in my opinion, more important than, than just testing

202. 37:08

     or evaluations. Even if you have, like, some online evals, IMO, you need to have, like, really, really good, uh, end-to-end monitoring of the entire system.

203. 37:18

     Right.

204. 37:18

     Curious if you have any thoughts there as well.

205. 37:19

     So I think another major issue is, like,

206. 37:24

     the unknown issues are-

207. 37:25

     Yeah

208. 37:25

     ... even more important. So I think having, like, a generic user frustration classifier is actually really powerful.

209. 37:31

     Yeah.

210. 37:31

     Uh, say, for example, uh, we also have this another feature called, like, uh, issues, which basically is, like, a agent that sort of, like, mines for, uh, newly occurring issues, right?

211. 37:43

     Say, for example, Sentry has, uh, similar to Sentry in a way, where there's, like, a new exception which is occurring, so it, like, alerts you on that. So say, for example, uh, you are a coding agent provider, and then certain providers are, like, failing all of a sudden.

212. 37:59

     And then you can actually... It can actually figure out, okay,

213. 38:03

     there's a subtle spike in user frustration and sort of, like, similar to how a human operator would, uh, it can start digging into are there any patterns for the spike in user frustration?

214. 38:15

     And then it could figure out that, okay, people who are, like, say, for example, are dealing with, uh, a specific Postgres provider start to face issues. Uh, so we are actually seeing this happen live in, for a couple of our customers where they had a database provider failing, and then we ha- had, like, an automatic, uh, issue

215. 38:36

     being created for them.

216. 38:39

     Yeah. Basically, once you have that good set of signals, like a good user frustration classifier, as Danny said, you can basically do clustering on it to find, like, what are, what is the root causes.

217. 38:51

     Um, yeah.

218. 38:55

     Um, how do you integrate with Light Elements or D or what does that look like?

219. 39:00

     Do you wanna talk about integrations?

220. 39:02

     Yeah. I think we might have a very basic SDK for it. Uh, so our Python side of support is, like, fairly weak right now. Uh, but we have a fairly good, like, AI SDK support built in.

221. 39:15

     So the AI SDK even has, like, self-diagnostics built into it. So we inject the tool for you, uh, so that you wouldn't have to do anything. Uh, but it is going to get better, so

222. 39:28

     yeah. Yeah, we actually released, like, 10 different SDKs in the past month. So we have a person [chuckles] working on SDKs actively, so it's going to improve, yeah.

223. 39:39

     Um, I have a question about experiments, running experiments. You know, I'm curious how your platform helps me with this issue. Um, I have a team of, of around 10 people building my AI agents platform, and we constantly change things, like, all the time.

224. 39:58

     And we have a lot of feature flags, and some of them are experiments. And the, the, the rate of the change is, is so big. Like, every day, everything changes.

225. 40:09

     I just cannot, uh, uh, compare the, the, the, the traces, uh, the sessions of users because, um, I don't have [laughs] enough time to do it, you know?

226. 40:22

     Yeah.

227. 40:22

     Um, I, I need to have, like, a base system and then, uh, run, uh, a few days with, uh, on, uh, on parts of the users-

228. 40:31

     Mm-hmm

229. 40:31

     ... with one feature flag enabled so I can actually ex- uh, compare the data, analyze and get some insights out of it. And I just don't have enough time to do it, [laughs] you know?

230. 40:41

     Yeah.

231. 40:42

     Um-

232. 40:42

     So how are you doing-- How are you sort of doing that right now? Are you just kinda...

233. 40:46

     It's like Wild West, you know?

234. 40:48

     Yeah.

235. 40:49

     That's why I said I'm just using, mostly using Claude because I, I just give it everything and ask it questions and try to, like, figure out the, the insights.

236. 40:58

     But it's not really, you know-

237. 41:02

     Yeah. I get what, I get what you're saying. So a few things there. The first is, like, you can use experiments. If you, if you wanna keep that, like, shipping speed, you don't have to run, like, long multi-day experiments.

238. 41:12

     You can ship something, and if you have a sufficient sample size, you could see pretty quickly if there's any regressions or not. Like, you just... Maybe it's, like, 1% different or 2%.

239. 41:22

     That's enough for you to be like, "Okay, it's fine. It's not, like, breaking anything drastic." The other thing is, like, kind of what Danny was talking about. This-- We have an agent which is basically, you could think of, it's basically exposing all of these signals to Claude to make decisions on if things are better or not.

240. 41:41

     And we're thinking also about how we close this loop. Um, maybe you h- you have a really good set of signals, and then you, you have, like, essentially an agent that can look at all these signals, and then it can find issues based on that, what's changing, et cetera.

241. 41:58

     And then it can, like, create a PR based on that, and then it can see how, you know, run some new experiments based on these new PRs and bui- like, this can become this infinitely self-improving loop.

242. 42:09

     Um, which is like, which is very interesting, but that's one thing that I'd think about. I don't know if you have any additional thoughts.

243. 42:16

     It's slow because you need to deploy it to production and wait for, for some data to-

244. 42:21

     Yeah

245. 42:22

     ... to be gathered.

246. 42:23

     Yeah. Depends on how, how much data you have.

247. 42:26

     Yeah.

248. 42:26

     Um, but that's p- it's, uh, yeah, it r- it, it really depends on how big these sample groups are as well, et cetera. Sometimes it's, like, a few minutes you can tell, but sometimes you wanna wait for longer.

249. 42:41

     Yeah.

250. 42:41

     Uh, does your platform help with the, like, uh- Experiments, uh, on sessions, so we can maybe automatically enable-

251. 42:50

     Mm.

252. 42:50

     -some experiments. So, so we can, like, uh, take care of the logic that every session has only one experiment-

253. 42:55

     Yeah

254. 42:55

     ... enabled, so we can easily compare it to the base or something like that.

255. 43:00

     We have-- We're working on stuff like that acti-actively, but, uh, yeah, coming.

256. 43:03

     Great. Thanks.

257. 43:06

     I'm wondering whether we are storing any original traces, and then when I come in and I implement my new signal source called by whatever, if you can, like, fail-

258. 43:15

     Yeah

259. 43:15

     ... the signal for the original-

260. 43:17

     Yeah

261. 43:17

     ... produce, so I can do some kind of post-mortem analysis.

262. 43:20

     Yeah. We do. S-so you can, like, ingest all your historical data, and then when you create a signal, we actually sort of like run, like, a quick backfill of the past couple of days.

263. 43:30

     Yeah.

264. 43:30

     So, uh, yes. So that's definitely supported.

265. 43:39

     Thanks.

266. 43:40

     Is there a free plan to try it out?

267. 43:43

     Uh, we do have a free trial. We're gonna try to make... It's, right now it's two weeks. Uh, probably gonna make that longer soon. But if you just DM me, I can-

268. 43:51

     Yeah.

269. 43:51

     If you... Maybe I can have my... Should I-- Oh, do you wanna open this so I can just have our-

270. 43:57

     Yep

271. 43:57

     ... things? Well, yeah. But if you just...

272. 44:01

     So we are hiring. That's a, that is a, a thing, uh, that we're very excited about, like, trying to massively increase the size of the team. Um, and if you message me at either Twitter or you can email me as well, like, that's something I can just set you up with a longer free trial.

273. 44:19

     Yeah.

274. 44:19

     And, um, internally, so we use Opique and Sentry and others. Um, how do you guys... Like, I would imagine you guys also use those tools.

275. 44:31

     Yeah.

276. 44:31

     Yeah.

277. 44:31

     And how does Raindrop fits? From how I'm, I'm understanding it, it's you're creating signal with your own models, whatever you're using, so that-

278. 44:40

     Yeah

279. 44:41

     ... you make our lives easier to identify signals and, uh, like, harmful intents and user behaviors. Uh, but, uh, how would you maybe have an example of how do you have that full stack of how it works with the Sentry and Opique and, and, like, maybe where there are overlaps where you go directly for those competitors?

280. 45:02

     So if you're sending all the telemetry data, we can find any exceptions in the traces, tool errors, et cetera, and that's a thing that you can also track within Raindrop, and that's, that is a explicit signal.

281. 45:12

     So, uh, there's like implicit and explicit signals, so that becomes an explicit signal. Um,

282. 45:19

     do you, uh, do you wanna?

283. 45:21

     So I think most of the observability platforms will give you, like, the agent trace, the token usage, uh, if the tool call failed or not. Uh, but I think where we sort of like shine is sort of like the fuzzy part, the fuzzy failures, right?

284. 45:36

     Where the user is, like, frustrated, which I think matters more than, um, the explicit signal that you sort of get from Sentry. I mean, obviously, those are also important, but we focus a bit more on the fuzzier side of the failure space.

285. 45:51

     Uh, but at the same time, we also have a trace view. I think we also have a very interesting feature called, like, trajectories, uh, which sort of like visualizes...

286. 46:00

     If you want to find, like, uh, a trace, uh, which has, like, three different tool call failures, uh, so you can actually-

287. 46:08

     I think, well, maybe it's often, yeah, but-

288. 46:10

     Okay. Let me just get in. So you can sort of like describe the,

289. 46:18

     uh, type of trace that you want to look at. Uh, so

290. 46:24

     let's see. Hope we have, like, data, but, uh, you can more or less like describe, uh, the type of trajectories that you want to see, uh, instead of just, like, configuring it.

291. 46:36

     So we do both in a way, so you can obviously set up, uh, tools are failing sort of like alert as well.

292. 46:46

     So you can just search, like, for any trajectory. Um,

293. 46:52

     so yeah, you can see that this is sort of how the tools are be, being called, in what order. You can see which ones have, have errors. You click into them, you can see the input and the o-output to the specific tool, like what actually screwed up here.

294. 47:06

     Um, and you can see, okay, it's interesting that this has, like, this... The, th- No one lets you... Th-this is pretty much the only place where you can visualize tools like this.

295. 47:15

     Um, but you can see here, like, you can get a shape and understanding of the topology of what's going on here. And you can see when there's other ones that look similar, you can sort of see, okay, this kind of looks similar to this, and then that gives you a sense.

296. 47:28

     You can do, like, search on this. Again, we have an agent that can look through these and give you a sense of what's going wrong, and so it just makes it really easy to find, uh, issues in, in agents.

297. 47:40

     Um, yeah. Cool. All right. Anything else? No. Any other questions?

298. 47:46

     Can you export the data that you just showed?

299. 47:49

     Uh, the directories data?

300. 47:50

     Yeah.

301. 47:51

     Uh, what would you... You wanna export, like, just the raw trace logs? Or what, what do you want?

302. 47:56

     Uh, yeah, for the tool calls that are interesting.

303. 47:58

     Uh-

304. 47:58

     So I think we... Usually, what our customers do is that they already have, like, a OTel stream, right?

305. 48:04

     Yeah.

306. 48:04

     So we just end up being, like, another, uh, target, I guess.

307. 48:08

     Yeah.

308. 48:08

     But at the same time, uh, they do want us to, like, export signals that we label.

309. 48:13

     Yeah.

310. 48:13

     Yeah.

311. 48:13

     So we do support, like, uh, BigQuery and, uh, Snowflake. So we do export the event and then the signals that were classified for that event.

312. 48:23

     Perfect.

313. 48:23

     Last questions?

314. 48:24

     Can, uh, can we look at signals? Or like, um...

315. 48:31

     Let's do it a longer timeframe. So let's go over the last month.

316. 48:41

     So you can see stuff like refusals, and then again, if you click into any of these, you can get a sense of over time, task failure, jailbreaking, like what specifically is going on.

317. 48:50

     And then you have your self-diagnostics ones as well. Um, capability gap, et cetera. Cool.

318. 48:57

     And do you have open data on, like, number of traces that you guys know? Because it seems that with your clients, um, the tool is extremely valuable when you have those, um,

319. 49:10

     a-agents adapted at a very big scale. And so I can imagine that you have a lot of data. Do you have some data-

320. 49:16

     Yeah, it starts being... Is your question is, like, what's the smallest where it's useful or what's the-

321. 49:21

     Uh, like, like volume of, uh, all the data that you're receiving, processing, and generating signal around, like, um, how many jailbreaks do you see across all the clients?

322. 49:30

     Oh, do we have any sort of like-

323. 49:32

     Yeah, like, like how we've been-

324. 49:33

     We should, we should do something like that. That would actually-

325. 49:35

     Sort of guides.

326. 49:36

     That would be very interesting. We don't have anything like that, no.

327. 49:38

     I think there are, like, mixed opinions about that, I think. Eat Sleep does it in a way, right?

328. 49:42

     Yeah.

329. 49:43

     But people generally have a negative reaction to it.

330. 49:45

     Yeah.

331. 49:45

     Uh, maybe it's different here, but at the same time, do our customers want us to do that? Is, like, a different question as well, right? So.

332. 49:52

     Yeah.

333. 49:53

     Uh, but yeah, we would love to, but, uh, I think there are, like, compliance reasons where we can't actually put a customer's data out there.

334. 50:03

     Cool. Anything else? [audience applauding] All right. Thank you, everyone.

335. 50:10

     Thank you. [upbeat music]