Research to Reality: Bringing frontier ML research to production

Talk transcript

1. 0:01

   Hey, I'm Vaidas, and I'm a senior research engineer at Higharc on our labs team. So at Higharc, our labs team is basically our research and development arm, where we have machine learning researchers who kind of explore the frontier of AI ML, uh, and figure out ways to apply it to home building. [upbeat music]

2. 0:27

   Now, because we're in home building and we're in spatial reasoning, uh, we pretty much end up needing to use a lot of what's out there in AI. So computer vision to scan hand-sketched floor plans and parse them into our internal data model, um, reasoning agents to kinda carry the user through these agentic experiences.

3. 0:47

   We have custom transformers. Uh, we use diffusion models for image gen, uh, and basically a lot of kind of what's out there in AI ML, uh, we end up needing to use because of the kind of multidisciplinary nature of our product.

4. 1:06

   So hopefully that video gives you some idea of the research areas that we focus on that I've mapped out here, and that kinda leads us to, uh, the problem, which is in this, in this challenge of getting frontier research into production, uh, we need to start working with software engineers, let's say platform engineers, infrastructure engineers, back-end engineers,

5. 1:30

   who are very familiar with building robust and production-grade code, but are likely not familiar with the methodology and research in computer vision, in training your own LLMs, uh, and even some top secret, uh, topics that I can't reveal here. [chuckles]

6. 1:49

   Now, you kinda have the flip side problem with our, uh, ML researchers, who are very up to date with the latest papers and can, uh, pull together these concepts in novel and creative ways to develop new features.

7. 2:02

   Uh, but they have not really worked, uh, as software engineers typically, uh, where they've been responsible for production-grade, uh, APIs. So that's kinda what I wanna get into, is this, uh, this handoff, this, this baton pass of how do we facilitate that, and how do we do it productively?

8. 2:22

   And we look at this basically as a systems and process problem, and I want to kinda zero in on three main focus areas that, uh, you can use to, uh, improve, um, this, uh, the velocity of teams that, that are bringing research into production.

9. 2:42

   So the first thing we'll look at is research legibility. So let's say you have a, uh, ML researcher who's produced a, a prototype concept. How can they map that out, um, and make it digestible and understandable for these different software engineers and product managers who are gonna be jumping into the project?

10. 3:02

    Uh, the second is how to structure your code base. So we use a monorepo. Um, how do you arrange your code and modularize it so that it's ready to receive these new prototype concepts, um, and turn them around and stand them up quickly?

11. 3:18

    And then the third thing we'll look at is basically, you know, making the jump from a mapped out, uh, proven out research prototype that's gonna be going into this, uh, repo.

12. 3:29

    How do you decompose that prototype and stand it up on best practices in software engineering?

13. 3:37

    So taking a look at the first step, um, there's a really good article that, uh, that I wanna link, uh, a blog from the, The Pragmatic Engineer, where he talks about, uh, software engineering teams, uh, as they grow and start to scale, how important it is to write out technical design documents, uh, requests for comment, uh, whatever

14. 3:58

    you wanna call it, uh, specifications before building software, uh, as a way to align teams. So we have a very analogous document, um, that we require from, uh, all research prototypes that we call the research, uh, prototype taxonomy document.

15. 4:16

    So it's really just a technical design document from software engineering, uh, with some specific twists, uh, that, um, make it more, uh, digestible given its, its nature in machine learning.

16. 4:30

    So the first thing in that document, uh, we use Notion for this, but obviously any written document will work, is that we start with the kinda domain context and, like, what are the domain-specific, uh, you know...

17. 4:41

    We're in the architectural domain in home building, so what are these kind of novel, uh, ways to represent data? Maybe it's a Part T diagram. Maybe it's a graph to, to represent the kind of circulation graph through a home.

18. 4:55

    Maybe it's, um, embedding models or latent space representations. Uh, I like to say kind of picture a software engineer who just-- we just hired from JPMorgan. What are the kinda, uh, specific lingos and, and data representations that they might need to know before jumping into this project?

19. 5:14

    The second is kinda mapping out the business goal, right? What's the-- why does solving this problem matter, and what's the value, uh, in this ML tool? And then the four, uh, re-remaining parts of this document, uh, are, are pretty kinda conventional software engineering principles you might see in a TDD.

20. 5:32

    So, uh, the, the type safety, so we'll see later what our, um, machine learning repo looks like. But what is the type contract between our core product repository and this, uh, machine learning repo?

21. 5:47

    How are those types shared, and how do they stay in sync? Um, then kinda mapping out the, the persistence layer. Uh, is there a database? Uh, this is an area where, uh, we think it's probably best, uh, not to have our researchers spend too much time Uh, in the persistence layer, and just map out how far they

22. 6:07

    got, and this is a great first entry point once we start bringing in software engineering, uh, help on the project. Uh, then kind of mapping out the, the overall system architecture.

23. 6:18

    Um, is this a workflow? Is this a chaining of workflows? Are there external LLM calls? Like, what is the anatomy? What is the kinda taxonomy of this, uh, research prototype?

24. 6:30

    And lastly, how are we gonna merge this? How are we gonna decompose it? Um, and we'll get into that more, uh, later as well.

25. 6:39

    So going back to these kind of three, uh, levers we can pull, the first is kinda that research project, uh, taxonomy document. Uh, the second, uh, I wanna get into are, uh, how we structured our code, um, and how we, um, serve our existing, uh, features.

26. 6:57

    So we basically have a, a separate, um, repository from our core product repo, and this is, uh, all Python-based. Uh, right? It's all AI, ML stuff. Um, and it's basically a monorepo of, uh, cleanly isolated and fully decoupled microservices.

27. 7:18

    And so let's say data-driven entity prediction is our, uh, custom transformer model. So that is able to opt to, um, kinda grow and be iterated on and be fully decoupled from a different research initiative.

28. 7:34

    Uh, and it's pretty much a one-to-one, uh, researcher to microservice, uh, ratio.

29. 7:41

    So we find that works really well. We have a kinda gateway that, that guards requests, um, and it's all in one Docker bridge network, so the core, um, consumers are clients in our, in our web application make kinda API calls to this gateway, which then routes them to the appropriate microservice.

30. 7:59

    Now, uh, looking a bit closer at these, uh, individual microservices, uh, we tend to build them in a pretty simple layered architecture, right? There's an API layer, the business logic, and the data layer.

31. 8:12

    And then we tend to have, like, some really, um,

32. 8:16

    really cleanly documented specs, uh, so that agents can navigate these repositories and help accelerate our ML researchers as much as possible. So yeah, kinda taking a, a another way to look at this layered architecture, we have kinda the core business logic at the services layer.

33. 8:33

    Uh, that might, you know, make external, uh, LLM calls to, to foundation models, or we might need to pull in our own, uh, machine learning model weights in CI/CD.

34. 8:45

    Then we wrap that business logic with controllers, um, then we put API routers around those and expose them in FastAPI applications. And each of those microservices is a standalone application.

35. 8:57

    And like I said on the last slide, it's not really the client that makes a direct call to this microservice. It kinda goes to that, uh, gateway first, which routes it to the appropriate microservice.

36. 9:08

    And then within each of those microservices, um, in the root rep- in the root directory, we have essentially metadata, build instructions, a Docker file on how to, how to build this application, uh, the, the kind of project dependencies, and then Poetry or, or uv, uh, lockfiles, um, as needed.

37. 9:27

    And then so I like to kinda map out this, um, anatomical arrangement of our kinda three microservices that we have into production. And we can kinda see these, these trends and, and consistent skeletal backbones of these projects, uh, and it's very easy to map them out and make sure that they're growing, um, along best practices in software

38. 9:48

    engineering. So that's basically it for our, for our monorepo. We have these kind of microservices, and then, uh, sharing basically, like, underneath all of that in the repository, we have some GitHub actions to, to build and deploy our automated d- test suites and kinda linting, formatting, and type checks, all the stuff you'd, you'd kind of expect.

39. 10:09

    Uh, we have some Jupyter notebooks that run on, on modal for GPU compute, um, as well as some additional ML studies. But really the way we look at it is we have this kinda tooling layer and even this kinda fun CLI, but these all just support our ML engineers in bundling up these microservices that we, uh, serve

40. 10:30

    in production. So that's kind of the, the RPT is the, is the handoff document between an ML researcher and additional software engineering, uh, talent. The monorepo is, is where that, um, research project is ultimately going.

41. 10:48

    And then the only remaining step is, well, how do we get there, right? How do we jump from one to two? And the third lever I wanna talk about is the kinda decomposition, uh, and PR review plan.

42. 11:01

    Uh, and we look at that really as a design problem, where we need to kind of, um, figure out how to slice and dice a, a large research, uh, monolithic prototype.

43. 11:12

    So here are just some images of, of, um, an, uh, platform-wide agent feature where, you know, we really studied what axes to slice, uh, and dice these projects up on, what that dependency graph would look like.

44. 11:26

    And then we use Graphite, um, for kinda stacked diffs to then, uh, decompose these large monolithic prototypes that have been proven out, um, and then get the right eyes on, on review to make sure that these are ready for production.

45. 11:40

    Uh, we really like Graphite because it allows for asynchronous review, right? I could be working on a PR all the way up here while a domain specialist is still reviewing a different PR.

46. 11:51

    And once we've kinda decomposed these PRs, uh, thoughtfully, we can start tapping, uh, subject matter experts throughout the organization on the specific slices or smaller tightly, uh, decomposed PRs, uh, that they need to look at.

47. 12:10

    And so again, there's, there's a lot of overlap between this initial research project, uh, taxonomy document, where once we've mapped out these layers, these arc- the architecture, what kind of persistence there is in the types, that tends to inform, uh, your decomposition strategy on how to bring it into the, into the monorepo.

48. 12:31

    So to wrap things up, I want to revisit these kind of three focus areas that you can use to evaluate how well your team is bringing research into production.

49. 12:43

    And I wanna talk about kind of some, some diagnostic frameworks to understand if you need to spend a bit more time and attention in any of these three. So in the first step with your research legibility, as your team starts to staff research initiatives with product people, with software engineers, with AI engineers, is it obvious where they

50. 13:07

    should concentrate their efforts? And is it clear, uh, what kind of tasks they need to pluck off to bring this prototype into production? If there's ambiguity there, you may need to spend some time revisiting this process.

51. 13:23

    In step two, your code repository. As you start getting ready to bring prototypes into your production grade code base, is it clear where these buckets lie to put in new code?

52. 13:37

    Are there templates and existing frameworks and patterns that you can mimic? Or is it possible that maybe you've started to outgrow that code base and that system architecture, and every time you bring in new research concepts, you're fighting these old abstractions and having headaches from limitations of your repository.

53. 13:59

    And then lastly, in the decomposition phase, are you able to consistently estimate the timelines and delivery dates for moving research concepts into your repository? Is it clear which subject matter experts you should tap for review and for productionizing this research?

54. 14:21

    Uh, and if you're having issues here, it probably points to upstream issues, either in, uh, how this research is being coordinated and handing off, or perhaps the code base that's hosting it.

55. 14:35

    Uh, anyway, that's about all I've got. I hope this was helpful. I love, uh, thinking about ways how our team can speed up how quickly we can bring research concepts into production.

56. 14:47

    And if you made it this whole way, you're probably interested in that stuff too, and I would love to compare notes, uh, and trade ideas anytime. Thanks again.