How Forward Deployed Engineering is done at Factory

Talk transcript

1. 0:00

   [upbeat music] This is the Forward Deployed Engineering track in case you're in the wrong room.

2. 0:16

   Um, as you already know, forward-deployed engineering is one of the hottest topics in AI. The most important companies on the planet are building out massive FTE teams, so think OpenAI, Anthropic, Google DeepMind, you get the idea.

3. 0:28

   Forward deployed engineering was pioneered by Palantir many years ago to embed really strong software engineers directly into their customers' orgs, uh, to implement, customize, uh, their platforms around the nuances of the real world.

4. 0:40

   So today, we brought in some amazing speakers from Anthropic, Cursor, Factory, Ramp, Decagon, and many more, uh, to talk about the current state of forward-deployed engineering, how it works at their companies, and where it's going.

5. 0:52

   Our first speako-- uh, our first speaker is Eno Reyes. He's the co-founder and CTO at Factory, which is building autonomous software engineering agents for enterprise teams. Previously, he worked in machine learning and software engineering roles at Hugging Face and Microsoft.

6. 1:06

   Let's give it up for Eno. [audience applauding]

7. 1:11

   Yeah. Hey, everyone. Excited to chat today. Um, and, you know, basically, I, uh, my hope is that at the end of this, you guys get a sense of some of the work that we're doing on behalf of our customers and with our customers, and the role of what we call a deployed engineer should hopefully be a little

8. 1:29

   bit clearer since I think that there are honestly tons of different models, um, for, uh, for how this should actually operate inside of an org. Um, and so I think that when, when we start, I, I, I do think that there are some nuances in sort of like the Palantir-era playbook.

9. 1:47

   Um, and I-- generally, the way that forward deployed goes is I, I see that there are lots of different takes on sort of where forward deployed sits within the org, how much it interfaces with the actual product team or the engineering team, how much work is done on behalf of the customers versus with them, and how much

10. 2:05

    work is done on code itself or basically, like, in the software system versus with the humans and sort of strategizing, right? And so, um, generally, I think the-- there's, um, uh, the-- in this older model, a lot of the way that software needed to be built was you needed to go and access that code base.

11. 2:23

    You needed to integrate directly into data streams or software or products that basically you could only access behind the curtain of the customer. And so if you were building something that was heavily integrated into their environment, yeah, you kind of had the need to send and sort of parachute in individuals into the org.

12. 2:41

    Um, but really, that has transformed over time into, uh, a role that sort of forks out, and you see a lot of people who are sort of, quote-unquote, "forward deployed engineers" or deployed engineers or applied AI engineers and, uh, it's always, it's always a little bit unclear.

13. 2:59

    Are they doing maybe professional services work on behalf of their customer? Are they transforming, like, the product around an individual customer? Are they just building entirely net new things in the customer's environment, maybe on top of your product?

14. 3:14

    Uh, and I think that the, the-- at least at Factory, we definitely do not want to be doing professional services work on behalf of a customer. So if a customer says, "I want to do a, uh, modernization of a code base, uh, and it's, you know, I just got quoted from all of the big consulting firms.

15. 3:32

    It's gonna cost this much. Could you do this consulting work for us?" Uh, our goal is not to go and actually do that migration on their behalf, even if we happen to be using our product, right?

16. 3:44

    Um, and that is because we don't think that that actually makes our product that much better. Uh, and ultimately, that is a great way to get, I'd say, a decent amount of revenue, but I don't think that that's the way that you can scale a business out, uh, enormously, right?

17. 3:56

    And so what we've done is we've instead said, "We need, uh, deployed engineers to be the tip of the spear of the product." Um, and I'm gonna do this and then go back.

18. 4:06

    Uh, but, but really when we say the tip of the spear, what we mean is that, uh, deployed engineers are basically the stream of information from our largest and most critical customers of the engineering leadership in that org, the on-the-ground tactical engineers, their thought process about how software development and AI is actually happening at that org, and

19. 4:27

    then flowing all of that information back into our product to then rapidly adjust our product in order to then fit into the customer's environment better, right? And so Factory really should be, when it gets deployed, and I'll talk about what Factory is in a second, but i- we want that to be effectively self-assembled inside of our customer's

20. 4:46

    environment, right? Uh, and then there's a lot of work that goes into understanding that customer's environment, what-- the flows that happen, uh, and ultimately the ROI story. And what Factory really is to our customers is a set of building blocks for building a software, uh, software factory, right?

21. 5:06

    And so when we say software factory, what we mean is there's this implicit process that every organization in the world sits on top of, where signals from the outside world flow in on one side, and those signals could be a lot of different things.

22. 5:19

    It could be customer conversations. It could be bug reports. It could be internal Slack or Teams conversations. It could be an executive saying, "We're going to build this thing," right?

23. 5:28

    All of these are signals. Some of them have higher weight than others. And those signals flow in, and humans, implicitly or explicitly, then choose to then prioritize, triage, and build plans around those signals.

24. 5:42

    Those plans are then converted, typically by software developers, into changes into some source of truth, a code base, an engineering system. Um, and as those changes are actually executed on, they flow through a validation stage where people maybe review the code, they QA, they assess the security implications.

25. 6:02

    They, uh, pass it through automated validation like SAST tools, linters, type checkers. And ultimately, when everything passes, they then ship and deploy. And what do you do with deployed monitored software?

26. 6:14

    Well, it, it generates more signals, right? So this implicit feedback loop is instrumented- Very poorly, to be honest, at most organizations. And if you're able to take AI and actually transform each of these stages of the pipeline and build an understanding of what the workflow looks like at your org from each stage to each stage, then you

27. 6:34

    actually can get to the point where you have a, a flow-through from signal to deploy that has no human intervention. Now, importantly, that does not mean that humans are not a part of engineering this system, right?

28. 6:46

    But it is that the flow of signal to deploy is uninterrupted by a human. Um, and that software factory concept is obviously not something that can just snap your fingers and it appears, right?

29. 6:57

    Instead, it requires an investment from the organization. We, we like to say this is built, not bought, right? But what the platform that we've built basically provides to people are the canonical one model independent agent harness that you need to do this because if you wanna build a software factory, if you choose to build that software factory

30. 7:16

    in a m- vendor-locked solution that has, like, one model available to it, uh, that is going to not only be expensive, but two, uh, there's open questions about model independence and, like, what is the role of the model provider in dictating what you can or cannot build with your software factory, right?

31. 7:34

    Um, and if you also don't own the traces, the data, everything that flows through your software factory, um, then you're probably gonna be in trouble as you start to want to evolve your software factory, right?

32. 7:45

    And so with Droid, the harness that we build, you not only have model independence, but you also have access to every piece of data that flows in, in and out of Droid alongside centralized governance and control at the enterprise layer to be able to dictate where-- what information flows where.

33. 8:01

    Um, you can air gap Droid if you want. Some of our partners, um, in, you know, the most secure, uh, environments, uh, think finance, healthcare, uh, gov, uh, they air gap Droid, and they run their software factories entirely contained.

34. 8:15

    Um, one of our deployed engineers jokes that you could run Droid in a submarine if you wanted to, and that's, that's honestly true. And so when we think about what the role of this deployed engineer is in that context, which I probably should have started with, um, you know, you really need somebody who can go in and

35. 8:32

    say, "I understand this new model of building software, and I understand the building blocks and the pieces. I can help enable building and constructing these software factories with your team, but I ultimately would like to, one, make it so that our product effectively, you know, one click self-assembled into your environment," which is needed when you have forty-five

36. 8:54

    thousand people, maybe hundreds of thousands of, uh, of, of engineers. Maybe you have tens of thousands of code bases. Uh, you, you've gotta self-assemble, right? You just can't manually install this level of, of complexity.

37. 9:07

    Um, and, and also on the sort of, like, end loop, why do all of this, right? I, I would argue that there needs to be an ROI or an outcome story that is extremely clear from the beginning so that you can say, "Well, we know every code change that flows through that gets AI code review, AI QA,

38. 9:25

    AI security analysis is maybe eighty-seven percent less likely to hit a bug." And what that means is that we can reduce our, our bug rate by X, that increases our customer satisfaction by Y, and that leads to revenue or growth or new business, right?

39. 9:41

    Something needs to flow from this software factory process to core business goals. And that often is a complex story that requires engineering knowledge, it requires business knowledge. Um, and so if those are the types of things that you think are interesting, um, that is what deployed engineers today are doing for us.

40. 9:59

    Um, I've sort of outlined it a little bit here, but that teach the model step is super important because most organizations do not have an autonomy maturity model. They do not have a roadmap.

41. 10:11

    They don't have a conception of what it means to truly build an autonomous software organization, right? Uh, I think a lot of people ask the question, what do the humans do in this world, right?

42. 10:21

    For us, we see an extremely clear role for humans in evolving, refining, and scaling software factories, right? So you basically, the engineers at a company go from directly manipulating software to directly maintaining and managing a system that builds software.

43. 10:38

    And that sort of, like, upgrade in the level of abstraction that you operate at, uh, is actually very difficult, and a lot of people, uh, uh, find it extremely challenging.

44. 10:48

    I would argue that, in fact, most people, even very thoughtful software engineers, will have a learning curve in trying to shift. Um, the people who I think are, are well suited for this are DevX people who have already been thinking about enablement of other developers.

45. 11:02

    Uh, I think product managers who want to become very technical very quick can become really great at doing this. Uh, and I think that generally, like people who are used to, um, to working on teams where, uh, high quality dev environments were a priority, you will c- you will get some of the canonical things necessary to enable

46. 11:23

    these agents to succeed. Um, I haven't really talked about this last one, which is design the workflows. Um, and I will get to that in a sec, but I, I think that when I say tip of the spear of the product, like keep in mind, I really do mean everything that is happening inside a factory.

47. 11:41

    So our product encompasses enterprise controls, the Droid harness, the workflows that run on top of it, the observability tools, the cost controls, the auto model routing, the quality of the harness.

48. 11:52

    Like all of these are pro- potential opportunities of improvement that you will discover when you work very closely in these varied or diverse orgs, like how to solve. Um, so

49. 12:05

    Making a code base agent ready, right? This is a very challenging thing to do. Uh, most organizations have some degree of consistency in how they've chosen to build deterministic validation loops inside of their company, right?

50. 12:18

    So your code base runs linters, type checkers, uh, it might run some security scans, and it's like check mark. Like, it passes or it doesn't. The end-to-end test, they pass or it doesn't, right?

51. 12:28

    Or they don't. Um, what agent readiness really is, is it's a measure of how many of these deterministic validation loops are present inside of your code base. Uh, when you have a huge volume of these feedback loops, uh, agents are able to operate for greater periods of time on more complex tasks without human intervention.

52. 12:45

    So we have, like, a product that we call M-Missions, which I'll also touch on in a sec. But Missions is basically an extremely elaborate harness built around the concept of working on extremely difficult knowledge work problems that are validatable, right?

53. 13:00

    And so the quality of the output of these very long-running harnesses of advanced agents is directly proportional to the degree to which you can validate their work. And so if you introduce the ability to validate at scale, then you introduce increasing autonomy to the org.

54. 13:16

    So what we'll look at is we have tools that help scan all of these things. But oftentimes, uh, the change is not so simple. Uh, for, I'd say maybe thirty to forty percent of the low-hanging fruit, you click droid, "Please fix all of this," and it'll go in and it'll fix it, right?

55. 13:30

    But for the other sixty percent, some of them involve workflow changes. Sometimes humans are not used to the degree of, I would say, like, nitpickiness of these automated systems.

56. 13:40

    Uh, and so you have to sort of be aware of the concerns, the, the humans. You have to think about, like, the way that, uh, people are currently developing systems and say, "How do we introduce some of these more extreme validation strategies without interrupting the dev flow of the humans who are involved in the work?"

57. 13:57

    Um, and, and I mentioned Missions because really, I think this is one of the more endgame of the agent era, at least pre-software factory era. Uh, but the more endgame of the agent era style harnesses, where it's simply a long-running harness that has almost no human intervention except for the planning stage, right?

58. 14:16

    Where you go in and you say, "I would like to have this very bounded task. I know that I want to solve this task, and here is what solving this task means.

59. 14:25

    I will now basically push a lever of inference until the task is complete," right? And so that is actually unbelievably competent at solving problems where, like, is complete, is verifiable.

60. 14:38

    So if you can frame any problem as the set of verification, uh, systems that need to validate it, then you can solve that problem with AI today. Uh, and we've seen this work on some pretty insane problem spaces, like migrating, you know, thirty, forty, fifty-million-plus line code bases, uh, fully autonomously.

61. 14:59

    Um, working on advanced, uh, like, deep learning strategies around biomed healthcare, uh, sort of problems. Uh, financial institutions that optimize, uh, equity research, where you can actually build models of different equities and sort of analyze and compare and, and build sort of a system that can then back prop and/or trade on top of the, those equities.

62. 15:22

    Um, like, it, it's mind-blowing to me every day what I, what I hear people are using with these tools, but it is not something that you can just download, install, and hit play, right?

63. 15:32

    It does require agent readiness. So if your code base isn't agent ready, you won't see any of the success of the most capable AI systems in the world today, right?

64. 15:40

    So this is why we want people to go in and help our customers and say, "Hey, look, you can solve this actually very difficult problem, but it is going to require a different form of investment than you were thinking."

65. 15:52

    Less so solving the problem, more so preparing the environment for verification of the problem. Um, and b-by the way, if you're familiar with how these models are actually trained, like, this makes total sense, right?

66. 16:03

    They're, they get dense reward when they get post-trained on all these complex tasks. Models need dense reward. These verification signals form the basis of that reward that they use to keep them on track over a long-term goal-directed problem.

67. 16:16

    Um, so I sort of mentioned this earlier, but, but I think that the, the core goal for us really is to say, if we can hand over a model to you of how this should, this transformation should go, then we should theoretically be able to say, "Let's do this in a couple of different places, and then let

68. 16:35

    your team actually scale this out across the company." Um, I always use the analogy of, if you're familiar with Walt Disney's Epcot, uh, the, the theme park. Uh, like, basically, that theme park was created...

69. 16:47

    Uh, originally, Disney wanted to create, like, a master-planned e-exemplar city. He said, "Look, if I can create a city that is the future city, then I can use that as a model to the rest of the world's cities, and they can develop entirely new forms of transportation and flourishing," and it became a theme park.

70. 17:06

    But what's interesting is that in that small example, um, a lot of other cities actually did cite some of the ideas that he was writing down and sharing about what, like, centralized urban transit should look like.

71. 17:17

    And now you have, like, some more contemporary cities built in the last fifty years that basically modeled after that toy example. Um, what we wanna do is we wanna make sure that we get some of that lesson that if you have a working example of a city of the future, of a code base of the future, um,

72. 17:34

    people are smart, they're clever. Humans will look at that and they'll say, "Man, that's really cool. Let's bring that to my part of the code base," right? But if you build too much of an advanced example, then people will say, "That's a theme park.

73. 17:48

    That is not at all how the rest of the world works. I just can't see how that would apply to the way that we currently work today," right? So it's kind of a delicate balance that you have to walk of building something that demonstrates the future is achievable enough, but ultimately does not scare away, uh, an org

74. 18:05

    who is thinking, "Man, what is going to be the cost of transforming at this pace?" Right? Um- I always think about that quote, you know, "The, the future is here, it's just not evenly distributed."

75. 18:15

    Um, there are some code bases, and I, I say code bases, not even companies, that are truly remarkable. They are effectively, uh, beginning to run on autopilot. Uh, we ourselves have roughly fifteen to twenty percent of what we call, like autonomy, and our autonomy ratio is, like in the upper eighty percent, which means the ratio of actions

76. 18:34

    done by humans to AI systems before interruption, right? So our own code base is fairly agent-ready, pretty autonomous. Um, but, uh, the c-code bases of some of our customers are actually even more autonomous because they operate in more constrained wor- uh, ways, right?

77. 18:49

    So it's, it's sort of like a, a... It is not obvious, like who gets a hundred percent autonomy first. I would argue it's probably very contained internal tools. Like we have something we call like Legal Droid, which is our legal workflow.

78. 19:02

    That is effectively a hundred percent autonomously maintained. But our, like s- core harness, uh, we do not yet have validators that can validate some of the hard visual problems of a, like terminal-based harness.

79. 19:14

    Uh, things like flickering are really hard to catch, uh, in a verif- in a verifiable way, so we're unable to close the loop on some of those challenges. It's an engineering task to build a system that can verify some of those very hard problems.

80. 19:28

    Um, and that might give you a picture into sort of like the weird world of the future, where humans are sort of visually... Our, our advantages in being visual, our advantages in having context of the outside world, um, provide us a lot of work to do, uh, in order to build these systems.

81. 19:42

    So who's great at this? W-- I-- If you are a former founder, for sure you should do this. I think it's like the quickest way to basically build out...

82. 19:51

    I mean, each, like stage of the SDLC that Droid has, we think is a billion-dollar business. Like just code review, just incident response, just QA, just testing, like each of these, uh, you will help define basically the nature of these products.

83. 20:07

    Um, if you are someone who is used to tech or, uh, communication, right? If you are fluent in AI, you understand how to speak to every level, you have business acumen, you have executive presence, that is another great example of someone who should do this.

84. 20:21

    And if you're a systems thinker, if you love designing systems, if you love closing loops, modeling data, and understanding how the flow through a potentially extremely complex org should look, then you are also someone who would, uh, thrive at doing this.

85. 20:35

    So if all of this seems interesting, hopefully, uh, it does, uh, please do reach out. Um, and you can reach out to me directly. I'm just... Yeah, I'll say it.

86. 20:46

    It's on the slide. I'm [REDACTED:email_address], um, and so you can just email me directly. Um, or you can apply on our careers page. It's called Engineer,

87. 20:56

    Deployed. Uh, so that's the role. Uh, hopefully, this is interesting, and it gives you a taste of what we're doing at Factory. [audience applauds] [outro jingle]