From Signal to PR: Anatomy of a Self-Improving Agent

Talk transcript

1. 0:00

   [on-hold jingle] Well, thank, thank you all. Um, let me just get set up here.

2. 0:16

   So, not just, uh, the founder of Arize, but, but I tend to build an incredible amount of stuff. Um, [ts]

3. 0:27

   let's see if we can get this going here.

4. 0:30

   Oh. Sorry, one more second. Um, so not just a founder here, but, but also a builder. And I do my best to, um, uh, to, to, to build agents, assistants.

5. 0:45

   Um, we have an agent in product. We have a, an agent in product called Alyx, and, uh, and a lot of... I think a lot of my experience is, is come from actually, um, trying to make the stuff work and work well.

6. 0:59

   Our first version of our, our own agent frankly sucked. Uh, it was many years ago, uh, probably two years ago. We're one of the first in the space to do it.

7. 1:07

   Um, and a lot of what we have built, uh, has come out of that- our own experience in building and building this agent. And, and Signal is kind of our, our next generation of this, which is trying to automate a bunch of things which we do every day, uh, and build it into a product set that people

8. 1:25

   can use. Um, so I'm gonna try to-- I'm gonna go through this, this materials here. I'll try to go fast and try to show you a lot of product too.

9. 1:32

   I'm a product person. Um, so if you built a startup before, um, you, you've experienced this. Your platform's down. It's, it's late at night and you, and you wanna go fix it.

10. 1:44

    Um, and, and really the, the... It takes a lot of energy to go do that. And we're gonna talk about like the automation we built a little bit and, and what, what the future looks like.

11. 1:54

    And, and I truly believe, um, that the future of the, the observability space is, is actually changing massively right now. W-why is that? Well, observability used to be for humans.

12. 2:06

    It used to be a UI you click, a graph you click, something you look at. Um, and, and today it's-- I would argue it's a lot of 2.0, which is like this combination of coding agent.

13. 2:17

    So those of you who built skills, skills for, um, Pyroscope, Google Cloud, or, or whatnot, the, the... These, these skills help you with your, your human debugging these systems.

14. 2:27

    Um, and, and really telemetry is like this smoke, uh, thrown off of your system that can allow these agents to go make fixes. It tells you what path in the code it took.

15. 2:41

    Without that, you're guessing, and there's a million paths it could have taken. The, the data thrown off by your system allows, um, allows you to, to go, go, go use agents to go debug your software.

16. 2:55

    Evals add another layer to this. Um, but really what we're at here is, is how do I build systems that autonomously fix themselves? Really that, that is what we're after, both, both AI agents.

17. 3:08

    I put AI into my, my, my, my system. How do I have this thing just improve itself? A-and today we're kind of in the 2.0, which is a human making fixes and reviewing things.

18. 3:18

    Um, but there's a future we're all driving towards. And throwing off traces, throwing off logs, throwing off way more than you normally would and having agents run at this for a, a continuous loop is where we're going.

19. 3:31

    You can build at agent speed, but today you can't improve your systems really at this agent speed. So those of us feel this, this kind of governor happening within our, our, our, our products.

20. 3:41

    Um, and the, and the bottleneck is actually not the fix anymore. So those of us who've used these systems and, and use, use, um, coding agents with, with skills, the, the, the bottleneck's a lot of the, the confidence in, in do I have it right?

21. 3:57

    You know, a lot of this is, is about is this fix the right one to push? Um, and, and so these are kind of the challenges here and then how do you, how do you build this loop in a way that just moves faster?

22. 4:10

    Um, and, and a little bit of the way we, we've kinda come to do it and we do it in our system is we've kind of inverted this, this loop, which is like a human, you know, looks at things and, and an agent, uh, fixes it, to a person now can wake up [laughs] with, with an idea of

23. 4:28

    the issues based upon the errors occurred in their system. So, so the agent is actually... You know, maybe it's not a, a, a fix itself, but it's putting up an issue.

24. 4:36

    It, it's looking at the data before a human even looks at it. Um, and, and what you move from there is, is kind of humans grabbing tickets to, to having some amount of evidence, um, some deep evidence relative to whatever you're looking at already sitting in front of you by the time you actually even look at it. [clears throat]

25. 4:55

    And, and human review is kind of one thing, but it-- a lot of times maybe you're driving this little investigation a bit from where it started, so that's the reality of where we are today, is there's still maybe it's not human reviewing, but human driving the, the step two and three.

26. 5:09

    Um, but, but this is kind of what, what we view the loop as. And really what it is, is there's, you know, there's an event that occurs that you're kinda kicking things off on or you're looking at periodically.

27. 5:19

    Um, and then there's some context around that which is really driven by skills. Um, I guess a question for all of you, who's created skills in this room? Who's created a skill that, that, that interfaces to an observability platform?

28. 5:35

    Okay, handful. Okay, cool. Aw-awesome. Um, so the magic of, of, of skills that, that, that connect to observability platforms, um, is it can gather the context. The agent can decide what it needs, what it needs to look at, um, to, to start to troubleshoot what you have there.

29. 5:51

    Um, and then there's the idea of triggers, which are like periodic and, and, um, or and, uh, and event-based. And so the future observability actually looks a lot more like this than it does clicking around Grafana UI.

30. 6:05

    So first off, evidence will... Normally these, like, or, or, or what do you start with when you look at, uh, traces are, are pretty nice logs as well, uh, but, but, you know, most of the AI systems these days have, like, traces at the core of, of the agent framework.

31. 6:21

    So, so you kind of start with, with looking at traces, and this is, this could be periodic, you know, every five minutes. This could be based upon an event, an error.

32. 6:29

    And normally there's some combinations of these, which is, um, you know, some, some, like, uh, context and lo- you know, context and skills used to put together logs. Maybe there's the repo.

33. 6:41

    Uh, you want kind of a combination of all this together, um, to understand what to go fix. The repo tells you the code paths that, you know, the pa- you know, the, all, all, tells you everything that's there.

34. 6:51

    The, the production logs or traces that the agent pulls down, um, normally our skills actually pull, pull little temp files down into the, the repo, um, so that you kind of have this, this, this idea of what actually happened, what the code is there enough, and, and all that together to put up a fix.

35. 7:09

    Um, so it's this combination of the right data and file format in the repo along with your code in the repo that's kind of the magic of this. Skills which are composable for the agent to go actually put a f- a fix.

36. 7:21

    Um, and a lot of this, some of you, a lot of you probably do this locally today. You, you run this locally. You have an agent that you, you kick up.

37. 7:28

    Maybe you're spinning up, but it's on your laptop, and I think we all feel this, this, this move from, from this laptop, um, to, to maybe to, to basically sandboxes.

38. 7:39

    Um, uh, and, and, and really the sandbox is this, this running environment where, um, based upon an event or a periodic, you know, a periodic event, you can kick this thing off, and it does the same thing you were doing locally.

39. 7:53

    Get it working locally first, locally on your laptop, and then event-based based upon the observability platforms like ourselves. Um, you can trigger these on a schedule or, or kind of, you know, every, every error that comes up.

40. 8:05

    Um, and generally, you know, generally it's kind of put- putting the loop together to do this. And, and, um, and I wanna kind of give you one example. So this is Alyx, our agent.

41. 8:16

    This is a, a real example. It's a very simple one, and then I'm gonna show you what it looks like in product, um, this what, what we use every day.

42. 8:24

    Um, but this is just an example where, um, we had a, a, a stream canceled event. So, um, so Alyx is, is basically, um, Alyx is, is basically our, our in-product assistant.

43. 8:37

    Um, to-do update is, is a, uh, is, is a way of, of managing kind of its, its task list. Um, and it was trying to... You know, I'll, I'll walk you through the, the error in a second, but basically, um, it's calling a bunch of these to, to, to-do updates and kind of, um, errors out.

44. 8:52

    And, and so for us, it's, it's, you know, how do I put the data together, um, to debug this? How do I do it automatically? And, and Signal is just something that's running in the background for us that's putting up, like, issues relative to these things.

45. 9:05

    Um, this was, uh, a kind of one or two-line fix that it comes up with. These are, these are ideal, but a lot of times the fixes are bigger.

46. 9:14

    Um, and, and the bigger it is, the more likely a human's involved and kind of, like, spearheading it over the line. But again, it's about that, that cold start.

47. 9:22

    Can I start with, like, all this information on the issue and guide it the rest of the way is kind of where we are right now. Um, and for us, your job kind of moves from responder to reviewer.

48. 9:32

    Um, and, and, and the view is, like, traces and evals don't go away i- in any way, shape, or form. They're just, they're, they're a key part of the loop now.

49. 9:41

    You're gonna trace 10 times more. You're gonna log 10 times more because that helps you know what path your software took. Before you wouldn't do that because, because humans can't dig through all the logs.

50. 9:53

    It's just noise. But by logging and tracing more of your, like, is it every inch of your software? Maybe in some places. Um, by logging and tracing orders and orders of magnitude more than we do today, we can actually create these continuous loops that know what path was taking your software and, and, and actually have it fix

51. 10:13

    itself. So this is kind of my vision for where I think things are going, um, in a way. And, and for us, I'll show you Signal in a second, and you'll see all these...

52. 10:22

    I mean, I feel like there's, there's this-- think of this as an, an SR, you know, uh, something that helps you debug, maybe SRE for, for AI. Um, but I feel like there's a lot of black boxes out there, like, oh, there's a SRE agent that does this or SRE agent that does this.

53. 10:36

    All we're really trying to do ourselves is take your local debugging experience with Claude Code, Cursor, and run it periodically. So pick your sandbox, pick your harness, pick your skills.

54. 10:48

    We'll pre-bake a bunch of things with you. So we're just trying to, again, take the things we were doing locally and actually run them, um, uh, you know, run them in a system.

55. 10:57

    So we believe in, you know, an open approach, um, to this, and, um, I'll, I'll give you a demo of what this looks like, um, from a product perspective.

56. 11:07

    So, so this is, um, this is a, a financial trading agent. Um, given what you saw in the previous, uh, presentation, I would not recommend doing a financial trading agent.

57. 11:21

    Um, they, they, they, they're unlikely to make you money, uh, at least not, not yet. Um, maybe there's some people, uh, doing it good. But long story short is this one's, you know, uh, people asking questions about stock trading right now, and it's giving, giving answers.

58. 11:35

    Um, there's a lot of ways this, this can fail, and so this, this gives you... This is Arize. It's a platform. So first off, from a-- l- let me describe, uh, the, the products we have.

59. 11:45

    Uh, this is AX, which is our, our, our SaaS platform. Um, we also have Phoenix, which is open source, if you just wanna start tomorrow. Um, Signal right now is f- is just available in, in our AX SaaS platform, um, which also can be deployed VPC.

60. 11:58

    But, but this, this-- So, so give you an idea of our product lines, uh, if you wanna try out Signal, it's, it's in AX. Um-

61. 12:07

    And what it looks like is something like this, which is it's just periodically running and, and kind of coming up with, like, issues, and you can hook it up to your GitHub repo.

62. 12:17

    It can create an issue in your repo. You can create an evaluator from this. Maybe, um, maybe there's a, a specific problem by which you wanna catch again. You can add these to a dataset, so if you wanna add these...

63. 12:30

    And, and it has evidence associated with this, like traces, um, in this case. Um, in this one it has skills, like, for Google Cloud and some other logging systems, so we can front end, uh, a bunch of places the data.

64. 12:42

    We're, we're pretty good at building, I think, these skills to debug issues, again, uh, but you can add your own skills. So these examples here, you know, traces running out without a guardrail.

65. 12:52

    Um, there's, there's, um, uh, you know, safety, safety issues and intent issues. And a lot of this too is like, you know, how, how does this work? How do I...

66. 13:03

    You know, it feels a little too black box to me. Well, all this is open and open box, um, in the sense that, um, I can set up, you know, I can set up the harness that I want it to run on.

67. 13:16

    This one's Claude Code. I can pick my sandboxes and sandbox systems. Um, I can use Claude managed agents if I want. I can use Arize sandbox. Uh, why would I wanna use Arize sandboxes versus Claude managed agents?

68. 13:28

    Well, a lot of our customers, um, don't wanna connect their production systems to, to Anthropic. You know, you, you, you want your sys- you want these, these sandboxes to be debug your database or connect to it.

69. 13:40

    So we install in the VPC of a lot of, you know, big name companies out there, um, from, from Uber to, um, uh, to Bookings to you, you name it.

70. 13:51

    And, and these people don't wanna send their connections out, but they'll, they'll use the same, you know, uh, many, many companies,

71. 13:58

    um, are very comfortable installing a V- into a VPC and a- actually connecting it up. So you can use Arize sandboxes or you can use Daytona or any of, any of these that you're comfortable with, um, that you've built relationships with.

72. 14:10

    Um, and, and then from a, a, a platform perspective, you know, we support,

73. 14:18

    um, running... We support, you know, tracking the different agents that you're running, so you have this swarm of agents maybe you've kicked off. Um, maybe you're kicking off a Signal which, which is our agent that's running periodically.

74. 14:31

    Maybe you're kicking off your, you know, you've named another agent, um, in the system. And these all support, you know, viewing the session that, that ran, downloading the transcript, and you can resume a Claude session locally too.

75. 14:46

    So the idea is that this thing's constantly running, you're picking the harness, the sandbox. You're deciding the prompt. If you want, "Hey, don't be aggressive," or, "Look for, you know, look for security issues."

76. 14:56

    So you're deciding the prompts that drive this, and you're also deciding the skills that go along with this. So, um, in a preset here, um, I can add, you know, add different skills, I can add my own skills, I can link repos, I can...

77. 15:13

    I have pre-baked skills too. Um, so the idea is that observability platforms are really starting to get... are, are becoming tied to the continuous loop, to the, the fix, not just the, the, the signal.

78. 15:25

    Um, and, and you, you want to take your local experience you have debugging the stuff locally, you wanna take the evals that are running and, and actually have these all work in something that puts up a fix or at least gets you a cold start, and then I can take it over locally if I want to continue

79. 15:42

    debugging, uh, from, from here. Um, so this gives you, um, a rough idea of, of kind of, of, of Signal to PR, um, what we're doing. Um, I do wanna offer, you know, questions if people, people have any questions on what we're doing or how we see, uh, the industry evolving.

80. 15:59

    Happy to, happy to answer. Thank you. [audience applauding] Yeah, go ahead. [laughs]

81. 16:15

    Um, a version that... So I work at a SaaS platform company. We get a version of the question of like, why can't we just connect Claude Code to your data and have Claude Code do all of these things?

82. 16:28

    I think there's like a version of that question that can probably be asked for these autofixes, right? Like-

83. 16:32

    Yeah

84. 16:32

    ... why not have Claude Code read the traces and push the PR itself?

85. 16:36

    Yeah.

86. 16:36

    I'm curious how you would respond to that question.

87. 16:38

    Yeah. Uh, uh, so, so why wouldn't have Claude Code kind of hook to your, your data and just, just do it? Um, the answer is like, you, you should. [laughs]

88. 16:47

    Um, like, like the vision and what we do actually at, at Arize is we have, uh, a lot of skills. I think first off to, to make that really work well, you have to do a bit of well-designed skills.

89. 16:57

    In the data space, the skill, like the, the important things of designing these skills are, are around, are really around getting da- you know, finding the right data first.

90. 17:08

    So I wanna find a group of traces relative to a session or something. Getting that data into the repo in a file format, these harnesses are magical with files, so you get the file of what happened.

91. 17:19

    In some cases, we have 10 meg files like sitting in the repo. Um, so it's designing the skill to be really, really well done with the, um, with this data a- and then giving Claude enough skills to be composable to find the issues.

92. 17:31

    So the answer is absolutely yes. Like, we have Pyroscope skills that will find memory issues. We have facets in Pyroscope that the skill knows how to use. I can cohort by customer to see if a customer is causing an issue.

93. 17:43

    Um, a- but, but you've gotta kind of design the skill surface area in a way that Claude can really, really work well a- and, and it's not just like point Claude at the data.

94. 17:54

    Mm-hmm. I see. Yeah. Thank you. All right. Any other questions? Anyone else?

95. 18:01

    Oh yeah. Okay, one more.

96. 18:04

    Thanks for the talk. Um, there was quite a few mention of evals, but, you know-

97. 18:08

    Yeah

98. 18:08

    ... I'm, I'm looking at the traces, so I, you know, I, I understand the concept of traces, but where, where do the evals come in when you have that signal that says, "Hey, something broke in production"?

99. 18:17

    Yeah.

100. 18:17

     Or-

101. 18:17

     So, so the, so the evals typically will... The, um, the evals essentially are running and being layered on typically to the production traces, something we call online evals. Um, let me see if this one has an example here of it.

102. 18:32

     Um, so, so evals actually are data on the trace itself, and so the agent knows how to, uh, grab the data from traces, knows how to, um, visualize and, you know, the skills to basically pull data for, for the aggregate values of evals across the traces so that, so the, the skills that you give, uh, the harness

103. 18:52

     allow it to get the data on the evals from, from the traces. Um, so evals are kind of like... I view them as, at least, at least the first generation eval- evals, which are LM as a Judge, um, as a, as a AI layer that allows you to run periodically and, and assess your system, but it's like,

104. 19:10

     it, it, but it's adding a little bit more s- you know, pre-processed information on, on the data. That, that... And then as Signal's running, it's using data from the evals that were layered on, um, in addition to all the raw data that it has there.

105. 19:24

     Um, it, but it tends to be like you build an eval for a failure you've seen before a lot of times. So I have these prompt injection things that I'm trying to catch or something, or, um, or, or a failure in, in the way it's responded maybe to, to something before.

106. 19:38

     So they, they tend to be this like, you know, um, at least the LM as a Judge is it tends to be like this, this thing you, um, pre-set up.

107. 19:47

     And then you can actually create evaluators for failures. Say you find this failure that's pretty common and happening all the time, I can create an eval so I can catch it next time.

108. 19:54

     I just, you, you think it as like almost an, an AI, um, assessment that's always running. Uh, the other note is the LM as a Judge can run really at scale.

109. 20:03

     W- Every, you know, I have customers who, who lay, you know, layer LM as a Judge across, um, their full data set, uh, where, where this tends to be like, you know, um, more periodic on a lot of data.

110. 20:16

     So cool. Thank you. [outro music]