Emulated: The data for fully autonomous software engineers and companies

Talk transcript

1. 0:00

   [upbeat music] So I appreciate the intro.

2. 0:15

   Uh, my name is Joseph, and this is my co-founder, Sid. Emulated is a data lab focused on increasing the reliability and autonomy of AI agents. And if you've been an AI engineer, and you've watched the talks, seen the tracks, then there's probably one takeaway that all the talks have in common, and it's that we're headed towards a

3. 0:37

   future where agents are able to perform useful work over longer and longer horizons with little to no supervision. So today, we're gonna answer some of the questions of what this means for the data and model layers.

4. 0:52

   Uh, we're gonna touch on some pretty cool things, uh, so look out for them, um, like how to simulate a company within a sandbox or sandboxes for multi-node systems and distributed clusters.

5. 1:04

   Um, and if we have a little bit of time, we'll also go into some of the work that we're doing with post-training pipelines and how these new types of sandboxes are affecting post-training infra as well.

6. 1:17

   So where Sid and I come from, um, our backgrounds are in network infra, distributed databases, and sandbox infra, and these are all areas where the workloads are mission-critical. Um, we all saw a couple months ago that, uh, when do- something like DynamoDB goes down, so does us-east-one and half the internet. [chuckles]

7. 1:39

   Um, and working on these systems, we saw a model capability gap when it came to operating and building these systems as scale, [REDACTED:username] scale, and thinking about, uh, the consequences of architecture and sys-system design over the course of years.

8. 1:55

   Yeah, so it led to a pretty, uh, natural question, right? For such mission-critical services, why is it that my model or my agent is so proficient [REDACTED:username] handling the application layer but is-- struggles when it comes to reasoning through infrastructure complexities?

9. 2:12

   For example, things like MVCC on a database engine, which can lead to corruption issues, which is one of-- which was one of the roots of the DynamoDB failure a few months ago.

10. 2:23

    Yeah. So like with everything in NML, uh, the gap in models is usually a gap in data. Models typically are only as good [REDACTED:username]-- as data is. Um, and to really highlight this point, right, model capability has never, uh, regressed whenever you introduce more high-quality data.

11. 2:45

    Um, so with that being said, what is the data gap then? What does data look like right now, and how is this influencing the model capability gap here? So if you look [REDACTED:username] any of the frontier or recent benchmarks, like SWE-bench Pro, Terminal-Bench, or something like Frontier-Code and DeepSwe,

12. 3:05

    um, the tasks only operate within the code base. Uh, the agent is given a pretty large, uh, task, uh, and over the course of fifty to one hundred turns produces a couple-thousand-line PR.

13. 3:22

    Um, but it doesn't do all of the work that a human does. It doesn't do, uh, what a PM does with talking to customers, understanding their problems. What an engineer does with trying out different approaches, performing-- performance testing them, um, and owning the underlying infra for the code base over the course of not just months, but years.

14. 3:44

    And this is really the gap that we're closing. We've taken software engineering companies, and we've put them into containerized environments. So this includes, uh, include like organizational context, like projects, incidents, customer conversations.

15. 4:01

    Uh, the agent also has to deal with issues that only appear [REDACTED:username] scale, like network failures between distributed nodes, data corruption, and clock skew. And through all this, we also want the agents to reason about orchestrating through distributed clusters and also thinking about things like operational blast radius while solving live traffic.

16. 4:21

    And the result is that the tasks that these agents have to complete or we want the agents to learn is that environments are far more complex and long horizon than a simple code diff.

17. 4:35

    So let's just-- let's bring a picture into the mix because it tends to make things more interesting. Uh, here's an example we've built of an etcd consensus cluster that a typical production service might rely on.

18. 4:49

    So an early environment, uh, might, uh-- tended to operate and work primarily on that little blue square entitled etcd source code in the bottom right there. But a lot of the fun and the model capability gap that results from it is really in everything that surrounds it.

19. 5:08

    So you s- you start with the tickets, projects, postmortems. What are the train wrecks? Why did they happen? How did customers feel about them? And oftentimes, those aren't necessarily up to date.

20. 5:21

    Um, the agent has to incorporate all that when it's reasoning through the actual change that current environments have it make. After it makes that change, uh, you need to kick, kick off rolling deployments.

21. 5:35

    Those deployment systems can oftentimes be complicated, have conflicts, may not work. Um, and all through that, when you're finally migrating off of-- from old hi-hardware onto new hardware, um, you run into unforeseen problems which you did not sort of-- that, that, that the agent has to reason through in real-time, just like a, a human would, right?

22. 5:58

    You have, um- Failing nodes. You have stale deprecated nodes. And while all of this is happening, the service can't go down because there is a blast radius to serving live traffic.

23. 6:12

    You have to observe and monitor your service. All of these components in, in, in the system is really, uh, what sort of exemplifies, like, a full end-to-end infrastructure task.

24. 6:24

    So what Sid is describing here is an environment in a single-node sandbox where we're simulating, uh, a distributed cluster with multiple nodes, flapping nodes, lagging learners, um, in a single sandbox, and you can get pretty far with this, right?

25. 6:41

    Like, you can see that there's live traffic, there's a lot of operational issues that a real engineer would have to deal with, and you can make this pretty long horizon by just, say, doing multiple deployments instead of just one.

26. 6:54

    But really what we're seeing is that this is not enough. Uh, this fits into standard post-training pipelines in the sense that a standard post-training pipeline is kinda boring. Uh, it's kinda homogenous.

27. 7:06

    You know, everything just runs Harbor. Everything is a single sandbox, containerized. But real infrastructure, uh, doesn't work like this. Uh, this isn't how real companies run, and, uh, even though you can use something like deterministic simulation to simulate network failures, it doesn't represent what you might run into if you're building an AWS scale

28. 7:31

    service. So I did see, I think, a couple people [REDACTED:username] AWS, somebody had Viceroy open on their laptop. Um, fun times. Um, but let's imagine here that we are all AWS engineers, or GCP engineers, Azure too, no shade, right?

29. 7:47

    Um, and we are building a cloud service. Um, it can also be some infrastructure service like Datadog, Vercel, Supabase. Uh, all of these services run into the same problems.

30. 8:00

    You start off with a shiny piece of software, and this piece of software can serve a single customer pretty well. Um, maybe it's running on your machine. If you're working for NLB, this would be a load balancer, right?

31. 8:11

    If you're working for AWS Lambda, it'd be some sort of serverless runtime. But, uh, it needs to actually run somewhere. So if you're an infrastructure engineer, next step is you get into resource provisioning.

32. 8:25

    Um, and this is already where the single-node sandbox starts breaking down. How do you provision resources within a single sandbox? You can't exactly simulate something like EC2 or s- Cloud Run, right?

33. 8:37

    Um, so you get into this host provisioning. Uh, it also includes provisioning of other resources like VPCs, subnets, security groups. Um, and you need to expose this through some sort of API because your customers are gonna wanna do things like, "Oh, give me this shiny piece of software," or, "I don't want it anymore.

34. 8:57

    It costs too much. I'm going bankrupt. Delete it, please." Um, and so you're gonna need some sort of front-end API. And if you have enterprise-grade customers who really care about quality, then you're gonna have to meet certain bars like throttling, authentication, authorization.

35. 9:15

    You can't really, like, go without these things, right? Uh, if you're AWS, then that's CloudTrail too. Um, and then beyond this, uh, software is living. People forget this all the time, especially, like, investors, right?

36. 9:30

    Like, they'll be like, "Oh, you wrote it. You're done." Um, but

37. 9:34

    software is living, and you probably need some sort of software deployment component as well. Uh, something whenever you have an update to roll out, roll it out, um, and God forbid something goes wrong, roll it back.

38. 9:48

    Uh, you need to manage all the different versions and make sure deployments are gradual to limit your blast radius. And we're just kind of getting started with this. There's all sorts of things that you need to think about, like health monitoring with awareness for network partitions, uh, and then how do you communicate with your host so you

39. 10:05

    can change configs on the fly? Um, maybe your customer actually wants to call your endpoint, so you need DNS and cert management. And then, you know, your service grows a bunch.

40. 10:17

    You need to keep track of all your resources, what's going on, fraud and stuff. Then you need admin consoles, uh, telemetry, billing if you're making money, um, all sorts of things.

41. 10:29

    And with all of this ... I think there's, like, one more slide for-

42. 10:32

    Yeah

43. 10:32

    ... was it scheduling? Yeah. Um ...

44. 10:36

    I think the point is fairly clear [REDACTED:username] this point. Beyond, beyond a certain threshold, there is a critical mass [REDACTED:username] which sandboxing on a single node, uh, can only get you so far, and that's why we envision the, the future being, going towards a world where environments do provision real infrastructure.

45. 10:58

    Yeah. So what this is is, um, a multi-node sandbox

46. 11:04

    with access to real infra, real cloud resources. Uh, we kinda put a cloud in box, so cloud box could be another name for this. Um, and as you can imagine, changing the sandbox type so drastically here affects post-training pipelines as well which, um, I think we might be running a little bit low on time, so we won't

47. 11:24

    get, like, too much into it. Um, but yeah, like, uh, one really cool thing too is, like, you can put a post-training pipeline in the sandbox, um, and there's some cool stuff with model training and RSI that you can get into there.

48. 11:38

    Um, so, you know, then this begs the question, uh, this is all cool stuff, Joseph. Uh, thank you, Sid, for speaking. Why are you leaking all of this alpha, right? [laughs]

49. 11:50

    Why are you, like, telling all your organizational secrets and telling everybody, like, oh, okay, you know, how do you build a system like this? Um, it's because, uh, we're really interested in these challenges here.

50. 12:02

    We think they're very fun. Uh, um- You know, we think they're really cool. We think that you guys are cool people, uh, or maybe I'm just lying. Who knows?

51. 12:09

    Uh, and we wanna share these challenges with you, uh, in case you're interested in working on them as well. As you can imagine, there's a lot of different problems that we haven't touched on here.

52. 12:20

    Like for example, sc- spinning up the entire stack for something like AWS Lambda takes hours. [laughs] Um, how do you fit that into a post-training rollout? Uh, and then there's cost as well.

53. 12:31

    How do you efficiently manage this? How do you make sure the sim-to-real gap, even with real resources, it still exists, right? You still have to have live customer traffic.

54. 12:41

    You still have to have, uh, problems that only appear [REDACTED:username] a certain scale. So, you know, if you're a distributed systems engineer, um, you know, if you know this stuff, if you've trained models before, um, if you think that this stuff is cool, uh, then, you know, we'd love to talk.

55. 12:58

    Uh, we'd love to talk, uh, kinda like see where your opinions are, uh, hear what you've worked on. Maybe that's like, oh, Kubernetes, and you have, like, opinions. Well, everybody has, like, opinions on, like, auto-scaling and rolling deployments and whatever, but, like, really niche opinions, right?

56. 13:13

    Like [REDACTED:username] CD. Um, yeah, we'd love to talk to you and hear what you have.

57. 13:21

    Thank you. Yep.

58. 13:22

    Um, what's your, like, primary goal with Emulated?

59. 13:26

    Yeah. Um, that touches into why it's called Emulated in the first place, right? Uh, the real world is very, very complex. Um, and how we, as a industry, emulate the real world is incredibly contrived and low fidelity.

60. 13:41

    So Emulated's goal is really how do you make these agents own systems like this, um, maybe beyond systems, entire companies, by emulating the real world with full fidelity.

61. 13:56

    Yeah, go ahead.

62. 13:57

    Next question is, um, are you pr- predominantly focused on, like, infra and, um, DevOps, containers, you know,

63. 14:08

    um, effectively hardware kind of related stuff? Or are you also looking [REDACTED:username], like, full RL environments, like a digital twin?

64. 14:19

    You know.

65. 14:19

    Yeah, of course. Like, the question is, like, you know, infra is really cool. Um, in 2026, all of a sudden infra's come back, and it's very sexy. Everybody wants to work on it, right?

66. 14:28

    But there's other types of RL environments as well. Um, there is, uh, other workflows that you really wanna capture that aren't necessarily infra related. Uh, so the reason why we're starting with infra is, um, there's a couple.

67. 14:45

    One-- The first, most important one is it speaks to our background the most. Um, we think that domain expertise is something that informs how high quality your data can be, uh, especially with the boutique nature of data nowadays.

68. 14:58

    Um, and the second is that when we're simulating full companies, infra is the easiest approach. Uh, if you think about, like, any infra company out there, whether it be Supabase or Modal, uh, or any dev tools company,

69. 15:12

    the problem statement is pretty clear. Uh, engineers kind of know what they want. If you are working for Modal, you know that your users want a GPU sandbox, very low latency, very low cost.

70. 15:25

    You don't wanna h- fail halfway through your training run. Um, that's what you care about. So the problem statement becomes much easier, whereas if you are, you know, a company in the YC Summer 2026 batch, you're probably still trying to find product market fit, right?

71. 15:41

    Yeah. [REDACTED:username] the same time, there's also lessons learned that going really vertical on a single domain like infrastructure do translate into other horizontal domains. So we're also, um, exploring going deep into one and scaling out that way.

72. 15:59

    All right. Uh, really appreciate it. Uh, appreciate the questions. Um, we'll probably step out, and we can, like, take a couple more, uh, outside just to make sure the next speaker has room.

73. 16:10

    Um, yeah, uh, thank you guys for listening.

74. 16:13

    Appreciate it. [audience applauding] [upbeat music]