The Devops Engineer Who Never Sleeps

Talk transcript

1. 0:00

   [on-hold music] Hey, I'm Diamond.

2. 0:18

   I hope everyone's, you know, feeling the AGI today. Um, I'll be sharing our AI agents at Datadog and what we've learned building the DevOps engineer who never sleeps. I came all the way from the New York Times building, uh, right across here, uh, to see all of you, so I hope it's worth it.

3. 0:34

   I've been working for my entire career, about fifteen years or so, in AI, trying to build more AI friends than coworkers. Um, I wouldn't read too much into that.

4. 0:43

   I have human ones too, um, I promise. Um, throughout the kinda AI winters and lulls of the last fifteen years or so, I've managed to keep doing just that at Microsoft Cortana, um, building out Alexa at Amazon, working on PyTorch at Meta, and building my own AI startup that was working on a DevOps assistant.

5. 1:03

   Now, at Datadog, we're building out Bits AI, which is the AI assistant who's there to help all of you with your DevOps problems.

6. 1:11

   So today I'll, I'll talk a little about that, talk a little about the history of AI at Datadog, a little bit about how we think about AI agents today and where we think things are going for the future.

7. 1:23

   Datadog is the observability and security platform for cloud applications. There's a lot that we do, um, but it kind of all boils down to being able to observe what's happening in your system and take action on that, make it easier to understand, make it easier for us to, uh, simply understand and build out things to have a

8. 1:43

   safer and more DevOps-friendly system. We've been shipping AI for quite a while, actually. Um, it's not always in your face. It's not always out there saying, "Here's a big AI product," but things like proactive alerting, really understanding things like root cause analysis, impact analysis, and change tracking, and much more has been happening since twenty fifteen or so.

9. 2:07

   But things are changing. This is a clear era shift. I think of this kind of similar terms to the microprocessor or the shift to SaaS. Um, bigger, smarter models, reasoning and multimodal coming, uh, foundation model wars happening.

10. 2:22

    This general shift where intelligence becomes too shift-- too cheap to meter.

11. 2:27

    And what this means is products like Cursor are growing, you know, terribly fast, um, and really people are expecting more and more from AI every day. Um, with these advancements at Datadog, we're really trying to rise to meet this shift as well.

12. 2:42

    The future's uncertain. This kind of ambiguity creates opportunity, but there's a lot of potential.

13. 2:49

    For us, that's kind of the dawning of this intelligence age. We're working to move up the stack to leverage these advancements and give even more to our customers by making it so that you don't use Datadog as just the DevOps platform, but also as AI agents that use that platform for you.

14. 3:06

    This requires work in a few key areas that I'll talk about. Developing the actual agents, doing eval, you just heard a lot about eval, we think about that every day, for better or worse, um, and building out new types of observability.

15. 3:21

    There's a few agents that we're working on right now in private beta. The first is the AI Software Engineer. This kinda looks at problems for you, looks at errors, tries to recommend code, uh, that we can generate to help you improve your system.

16. 3:33

    The second is the AI On-Call Engineer. This wakes up for you in the middle of the night, does your work, hopefully makes it so you have to get paged less frequently.

17. 3:41

    And then we have a lot more on the way.

18. 3:45

    So I'm gonna talk a little bit about the AI On-Call Engineer first. This is the one that, you know, everyone wants to save them from that two AM alert.

19. 3:53

    You don't wanna have to wake up in the middle of the night, go and look through your runbook, go and figure out what's going on if you can help it.

20. 3:59

    Our On-Call Engineer is there to really make it so you can keep sleeping. This agent proactively kicks off when an alert occurs and works to first situationally orient, read things like your runbooks, grab context of the alert, and then goes and, you know, figures out the kinda common stuff that each of you would do on Datadog already.

21. 4:17

    Look through logs, look through metrics, look through traces, and kinda act in this loop to figure out what's going on.

22. 4:25

    The On-Call Agent's great for both automatically running investigations for me, but also, you know, being able to look through and find summaries and find information for me before I even get to my computer.

23. 4:35

    So if I want to get insights into why an alert just occurred or figure out why a trace might, uh, be showing an error, this agent can jump ahead, pull information for me, and show it to me.

24. 4:46

    We also have added a new page that makes it easy so that you can have human/AI collaboration. This is still something I'm thinking about a lot, is like what, what kind of collaboration do we expect?

25. 4:55

    We want our agents to act as humans, but we also need to be able to verify what they did and be able to kind of look over what they're doing and really learn from it.

26. 5:04

    It also helps you to kind of earn trust along the way. I can see the reason why, uh, this hypothesis, for example, was generated, I can see what the agent found, and I can make decisions about whether or not I agree along the way.

27. 5:19

    It also tells you things like what steps did it actually take out of your runbook? And kinda like a junior engineer who does this work, I can go ask follow-up questions, find out why it did a certain thing.

28. 5:30

    A little more insight into how we're making this happen. Much like a human SRE or DevOps engineer, our agent works to put together hypotheses on what might be happening and reason over them, coming up with ways to test them, use tools in the toolformer sense to try out ideas, run queries against logs, metrics, et cetera, and work

29. 5:48

    to validate or invalidate each hypothesis. In the case that it does find a solid root cause, our agent cau-- uh, can suggest remediations along the way. Again, just like a human might.

30. 5:59

    Might say, "Hey, we should page in that other team that's involved here."

31. 6:02

    Or it might offer to scale up or down your infrastructure. Over time, we plan to add more built-in actions and eventually discover new types of workflows based on what your team has done.

32. 6:12

    But if you already have certain workflows that you've set up in Datadog, um, we can tie directly into them and make it so that our agent can understand those workflows and how they might map to helping you remediate a problem.

33. 6:26

    And if it's a real incident, the on-call engineer is not usually done once an issue is remediated. You usually go and write a postmortem, you go try to learn from it, you share it with your team.

34. 6:35

    Our agent can do the same, write out your postmortem for you, look at what occurred at-- during the entire time, what it did, what humans did, and put that together so that you have something ready in the morning.

35. 6:46

    So that was the on-call engineer. Um, that's the one that is, you know, trying to help you in the middle of the night, trying to help you every time alerts come on.

36. 6:54

    Um, we also have this AI Software Engineer. I think of this as the proactive developer, the DevOps or software engineering agent who observes and acts on things like errors coming through.

37. 7:04

    This is kind of the error tracking assistant. It automatically analyzes these errors, identifies causes, and proposes solutions. Those solutions can include generating a code fix and working to reduce the number of on-call incidents you have in the first place, so they can work in concert to make a better system over time.

38. 7:21

    In this case, the assistant has caught a recursion issue, proposes a fix,

39. 7:27

    and even creates a recursion test so that we can catch it if it happens again in the future. We have the option to create a PR in GitHub or open the diff in VS Code for editing.

40. 7:37

    This workflow significantly reduces the time spent by an engineer manually writing and testing code and greatly reduces human time spent overall.

41. 7:46

    So what have we learned building out these agents and some of the new ones that we're working on today? Well, we've learned quite a lot. Um, there's a lot of things that we started with that we kinda went back and, and redid.

42. 7:56

    Um, but a few areas I'll touch on that I hope help you as you develop your own. First is scoping tasks for evaluation. It's very easy, you know, to build out demos quickly, much harder sometimes to scope and eval what's occurring.

43. 8:09

    Second is building the right team who's ready to move fast and deal with the ambiguity that comes with these kinda problems. Third is that, you know, the UX of old is changing.

44. 8:18

    Um, that's something that everyone needs to be comfortable with. And fourth is observability matters. You know, uh, um, surprising for Datadog to say that, I'm sure, but observability is terribly important even in this new era.

45. 8:31

    So scoping the problems, scoping the work to be done. I like to think about this as defining jobs to be done and really kinda trying to clearly understand step by step what you'd like to do.

46. 8:42

    Think about it from the human angle first, and think about how another human might go and evaluate it. Um, this is why we build out vertical task-specific agents rather than building out generalized agents.

47. 8:52

    We also want, where possible, this to be measurable and verifiable, and at each step. This has honestly been one of the biggest pain points for us, and I think this is true for many people working in agents, where you can quickly build out a demo, you can quickly build something that looks like it works, but then it's

48. 9:06

    very hard to actually verify that over time and improve it. Um, use your domain experts, but use them more like design partners or task verifiers. Don't use them as the people who will go and kinda write the code or rules for it, because there is a big difference in how these kinda stochastic models work versus how experts

49. 9:23

    work. You know, everyone kinda knows Noam and his, uh, anti-NLP, um, rants, but that kind of stuff happens pretty frequently to domain experts.

50. 9:33

    Eval, eval, eval. I can't stress this enough. Um, start by thinking deeply about your eval. The number of mistakes we made by not thinking about eval first is, uh, frustrating and something that I think everyone should think about.

51. 9:45

    It's very easy to build these demos, as I said, um, but everything in this fuzzy stochastic world requires good eval, even something small to start. This means offline, online, and kinda living eval.

52. 9:57

    Have end-to-end, uh, uh, uh, tasks, have end-to-end measurements. Um, make it so you also instrument appropriately the way to know if humans are using your product right and giving you feedback, and then make this a living, breathing test set.

53. 10:12

    Building the team. Um, you don't have to have a bunch of ML experts. There aren't that many to go around right now. Um, what you really need is you wanna seed it with one or two and then have a bunch of optimistic generalists who are very good at writing code and very willing to try things out fast.

54. 10:27

    Um, I'll also note that UX and front-end matters more than I'd like as a back-end engineer myself, um, but it's terribly important as you collaborate with these thing-- uh, with these agents and assistants.

55. 10:37

    Um, and then you want teammates and people who are excited to be AI augmented themselves. This is day-to-day U-- AI use. This is explorer types who wanna learn. This is a field that's changing fast.

56. 10:49

    Um, and if you don't have people like that, you're gonna kinda get stuck.

57. 10:53

    You want folks who kinda, you know, yeah, yearn for the vast and endless AI capabilities, right? Um, it's a big world out there and there's a lot going on.

58. 11:01

    Ye olde UX. Um, this is one of those things that I still, you know, we think about, we go back and forth every day. Um, it's an area that I didn't realize was quite so important initially when I started working in this field.

59. 11:13

    Um, despite my engineering sensibilities and lack of UX, it's terribly important. Um, this is such an early space of work. This is kind of one of the more important things here as you collaborate and work together.

60. 11:24

    But the old UX patterns are changing. Be comfortable with that. Um, and so far, I'm partial to agents that work more and more like human teammates instead of building out a bunch of new pages or buttons.

61. 11:35

    So who watches the watchmen, right? Um, you have these agents running around. Um, observability is actually really important, and don't make it an afterthought. Um, these are complex workflows.

62. 11:45

    You really need situational awareness to debug problems, and this has saved us time a lot as we start to work with, um, a new view that we're calling LLM Observability in the Datadog product.

63. 11:55

    Um, Datadog in general has a full observability stack. As mo-- many of you know, we can look at GPUs, um, we can look at LLM monitoring, we can look at really your system end to end.

64. 12:05

    But tying in the LLM observability has been very helpful because you have a wide variety of interactions and calls out to models you're hosting, models you're running, maybe models you're using through an API, and we can make 'em all, uh, kind of grouped together in the same pane of glass, so you can look at them and debug

65. 12:20

    what's occurring. I will note though, that this can get messy fast with agents. Our agent, for example, has very complex multi-step calls. You're not gonna look at this and figure out what's going on right away.

66. 12:34

    Um, this can be hundreds of calls. This can be, uh, you know, uh, tons of different places where it's making decisions about tools, looping time and time again, and if you just look through a full list of these things, you'll never really figure out what's going on.

67. 12:46

    So here's a quick, you know, sneak peek into a more agent view of what's occurring inside of our observability tools. This is our agent graph. Um, really what this means is that I can kinda look at it just like our agent did and looking at workflows that are occurring.

68. 12:59

    You can see in this, even though it's a big graph, uh, there's a little bright red node here. If we zoom into that, we can actually see where errors were occurring.

69. 13:06

    This is very human-readable, something that makes it super easy to figure out what's going on when your complex workflow is running.

70. 13:15

    As an aside though, I do also want to note what I think of as kind of like the agent or application layer bitter lesson. Uh, general methods that can leverage new off-the-shelf models are ultimately the most effective, um, by a large margin.

71. 13:28

    Um, I hate to say it, but, like, you sit there, you fine-tune, you do all this work on, like, the specific w- uh, you know, project, the specific task, and then all of a sudden, you know, OpenAI or someone comes out with a new model, and it handles all this, you know, kinda quickly.

72. 13:40

    A lot of the reasoning is solved for you. Um, we're not quite there where it handles all of it very quickly, but you should be at a point where you can easily try out any of these models, um, and don't feel stuck to a particular model that you're, you've been working on for a while.

73. 13:53

    You know, a rising tide lifts all boats here.

74. 13:59

    Um, I also think a lot about not just building agents, but what it might mean for other agents to be users of Datadog and other SaaS products. Um, there's a good chance that agents surpass humans as users in the next five years.

75. 14:11

    Um, I'm probably somewhere in the middle on my estimate there. You know, there are people who will tell you that'll happen in the next year. There are people who will tell you, you know, it'll happen in ten years.

76. 14:18

    I, I think we're somewhere around the five-year mark. Um, but th- this means that you shouldn't just be building for humans or building your own agents. You should really think about agents that might use your product as well.

77. 14:28

    An example of this is, like, third-party agents like Claude might use, you know, Datadog directly. I set this up with MCP relatively quickly. Um, but any type of agent that might be coming in and using your platform, you should think of the context you wanna provide them, the information you wanna provide about your APIs that agents would

78. 14:44

    use more than humans. So looking ahead, um, the future's gonna be weird, it'll be fun, uh, and AI accelerate-- is accelerating each and every day. I strongly believe that we'll be he-- able to offer a team of DevSecOps agents for hire to each of you soon.

79. 15:01

    You don't have to go and use our platform directly and integrate directly. Ideally, our agents will do that for you, and our agents will handle your on-call and everything like that for you.

80. 15:09

    Um, I also do think that AI agents will be customers. Many of you building out SRE agents and other types of agents, coding agents, should use our platform, should use our tools, um, just like a human would, and, uh, we can't wait to see that.

81. 15:23

    And generally, I think that small companies out there are gonna be building, built by someone who can use automated developers like Cursor or Devin to get their ideas out into the real world, and then agents like ours to handle operations and security in a way that lets, you know, an order of magnitude more ideas make it out

82. 15:39

    into the real world. Thank you so much. Um, please reach out if you're building any agents that wanna use us, um, or if you'd like to check out our agents as well.

83. 15:48

    Um, there's a lot to build here. And if you wanna work in this space, we are hiring more AI engineers and people who are just excited about it. But thank you very much. [upbeat music]