Lessons from Building LinkedIn's GenAI Platform

Talk transcript

1. 0:00

   [on-hold music] Uh, it's my pleasure here to share our journey on building out LinkedIn's GenAI platform.

2. 0:23

   Uh, my name is Xiaofeng Wang, ad manager of GenAI Foundation.

3. 0:29

   Uh, let me try it one more time. Cool. Uh, in today's talk, I'd like to first share our journey on building out this platform, especially on why we're building it, how we build it, and what we're building it.

4. 0:44

   After that, we will talk about, uh, some thought process on why this platform is critical for today's agent world. Um, hopefully after that, you agree with me this is critical component in your component-- uh, uh, in your company, and you also want to build this team.

5. 1:01

   I want to share some tips on how to build such a team, how to hire for such a team. Towards the end, we will share some key takeaways and lessons learned.

6. 1:11

   Before we dive into this application, uh, platform journey, I think it's important to first talk about the GenAI product experience because that's essentially what our platform is supporting for.

7. 1:26

   Back in twenty-twenty three, LinkedIn launched the first, uh, formal GenAI feature called Collaborative Articles. This is a kind straightforward, uh, GenAI feature if we are thinking in today's standard because it's a very simple prompt in, string out type of application.

8. 1:46

   Uh, we leverage ChatGPT-- Uh, I mean, we leverage GPT-4 model, uh, to create the long content, uh, articles on the platform and then invite our members to comment on it.

9. 1:59

   At this stage, our team helped to build some key component behind the scene, including the gateway to centralize the access to the model, uh, some Python notebook for the prompt engineering.

10. 2:11

    Uh, but at this time, we actually have, uh, two different tech stacks, uh, to serve the, uh, experience. In the online phase, we use Java, and in the back end, we use Python.

11. 2:23

    Uh, we wouldn't call this as a platform at this time.

12. 2:28

    Very soon we realize, uh, there are some limitation for this simple approach, especially it lacks the capability to inject our rich data into the product experience. Then in the mid-2023, we started to develop the second generation of the GenAI product.

13. 2:47

    Uh, internally, we call it Copilot or Coach. Here we're showing one popular such experience on LinkedIn right now. Uh, basically it looks at, uh, your profile and the job description and then, uh, use, uh, some RAG process to give you personalized recommendation on if you are a good fit to the job.

14. 3:10

    At this time, we started to build, uh, some platform capability. Uh, specifically in the center of our platform, we build the, uh, Python SDK on top of the popular, uh, LangChain framework to orchestrate LLM calls.

15. 3:27

    And it also provide the key value to integrate with our large-scale infrastructure, uh, in this SDK, so our developers can easily assemble an application. We started to unify the tech stack at this stage because we realized it's really costly to transfer the Python prompt into the Java world, not to mention the error during this process.

16. 3:51

    We started to invest on the prompt management or prompt source of truth. This is a sub-module at this stage, uh, to help developers to version their prompt and to provide some structure around their meta prompt.

17. 4:07

    Uh, the most important piece I'd like to call out here is conversational memory. Uh, this is, uh, infrastructure to help to keep track of the LLM interactions and retrieval content and then inject those content into the final product.

18. 4:24

    It will help us to build this kind of conversational, uh, bot.

19. 4:31

    Now, uh, zooming to this year, uh, actually in the last year, uh, we launched our first ever, uh, real multi-agent system called, uh, LinkedIn Hiring Assistant. Uh, this is, uh, multi-agent systems to help our recruiters to do their work, uh, efficiently, especially it automate several tedious task, uh, normally recruiter need to do manually, like,

20. 4:55

    um, uh, post the job, um, and evaluate hundreds of candidates, then, uh, reach out to them.

21. 5:06

    Our platform also start to evolve into the agent platform. Uh, from the framework side, we extend the support of the Python SDK into a more large-scaled, uh, distributed agent orchestration layer.

22. 5:21

    It will handle the distributed agent execution and also handle the more complicated scenarios like retry logic and the traffic shift. Uh, for folks who build agent, uh, I think you probably know the skills or APIs are one key aspect of the agent because we expect, uh, this agent to perform some action.

23. 5:45

    One investment we did at this, uh, time is around the skill registry. Basically, we have a set of tools, uh, to help our developers to publish their API into this centralized skill registry.

24. 5:59

    This skill registry can handle the skill discovery problem-

25. 6:03

    Skill invocation problem. So in your application, it's actually very easy to call the API to perform some task.

26. 6:11

    Another key component, uh, we invest at this stage is on the memory. In addition to the conversational memory, we extend, uh, its capability into the experiential memory. Essentially, it's a memory storage to, uh, extract and analyze, and infer the contextual knowledge from the interaction between the agent and our user.

27. 6:33

    We also organize this memory into different layers, including the, um, uh,

28. 6:40

    working memory, long-term memory, collective memories. Uh, this can help our agent to be aware of the surrounding content. Uh, lastly, at this, uh, time, we also realize the operability is super important because agent, uh, one key aspect to define agent is autonomous, right?

29. 6:59

    Uh, because agent can decide what API they can call, uh, what LL, uh, uh, LLM they need to call. So it's actually very hard to predict its behavior. So we started to invest on the operability.

30. 7:13

    Uh, particularly, we build our in-house solution on top of the OTEL to keep track very low level granularity of the, uh, telemetry data. So we can use this data to replay the, uh, agent call.

31. 7:27

    And we also add a actual layer of the analytics on top of it. So we can use that to guide the future optimization of our agent systems.

32. 7:37

    Let's put together all the components we build for this platform. Uh, we can classify them into four layers, basically, including the orchestration, prompt engineering, tools and the skills invocation, content, and the memory, uh, management.

33. 7:54

    Uh, of course, that's not everything in the LinkedIn GenAI ecosystem. Uh, in addition, we have our sister teams to build out the modeling layer, like fine-tune the open source models, responsible AI layers to make sure the agent is behave according to our policy and standard, and also the, uh, AI platform or machine learning infrastructure team to host

34. 8:19

    those models. The key value proposition for this, uh, GenAI platform is actually to, uh, be the unified interface for this complex ecosystem.

35. 8:34

    So our developers don't need to necessarily understand all those individual box when they build, uh, their application. Instead, they can leverage our platform to quickly access to this entire ecosystem.

36. 8:47

    Uh, for example, uh, in our SDK, the developer can just switch one parameter in the one line of the code to switch from the open AI model to our on-prem model.

37. 9:00

    Of course, they still need to do the prompt engineering, but that reduce a lot of the complexity on the infrastructure integration phase.

38. 9:09

    Uh, last but the most important is because of this is a centralized platform, uh, it provide a place to enforce the best practice and governance. So we can make sure our developers are building the applications efficiently but also responsibly.

39. 9:28

    As you can see from our journey, we actually started to build this, uh, platform piece by piece, and then this platform start to emerge. If we take one step back and think, oh, do we really need this platform at this time?

40. 9:41

    Especially there are lots of, uh, uh, uh, vendor product, uh, on this space. Shall we buy it, build it? And why do we need to buy it or build it?

41. 9:51

    Uh, here are some thoughts. Um, the short answer is yes. The reason behind it is, uh, we feel like, uh, GenAI is a totally different and new AI systems compared to the traditional AI systems.

42. 10:04

    So in the traditional AI systems, there's a clear cutoff between the, uh, AI model optimization phase and the model serving phase. So AI engineers and the product engineers can operate in two different tech stack.

43. 10:19

    Uh, they usually don't, uh, need to, uh, work on the same code base. But in the GenAI systems, what we're seeing is this line between the optimization phase and the serving phase disappear.

44. 10:33

    Basically, everyone is AI engineer who can optimize the overall system performance. This actually create the new challenge of the tooling and the best practice in the company.

45. 10:47

    Essentially, we think these GenAI systems or, uh, agent systems is a compound AI system. Here, we borrow the definition from Berkeley AI Research Lab. A compound AI system can be defined as a system which tackles AI tasks using multiple interacting components, including multiple calls to model retrievers or external tools.

46. 11:10

    As you can see, this is actually skill across AI engineer and product engineer. And I believe this, uh, GenAI platform is trying to bridge this gap.

47. 11:23

    To summarize, uh, we believe this platform is critical for your success, mainly because it can bridge the skill gaps between those two group of engineers.

48. 11:34

    Okay. Let's say if you want to build this, uh, platform in your company and how to hire it is a frequent question, uh, I heard. Um, I basically look into, uh, my great engineering team and, uh, extract all the key qualifiers from those top engineers.

49. 11:54

    And, uh, I put all the qualifications here. Uh, the ideal candidate in this team is a strong software engineer, uh, who can build infrastructure integration. They have a good developer, uh, PM skills to design the interface.

50. 12:12

    Uh, ideally, they have the AI and the data science background to understand the latest techniques. They are the people who can learn from the latest techniques, but at the same time, they are hands-on.

51. 12:26

    Unfortunately, [chuckles] it's really hard to get those candidates. If you get them, uh, it's probably worth more than a unicorn. [laughs]

52. 12:33

    Realistically, we are making multiple trade-offs in the hiring. Uh, here are some principle, uh, we follow, and it's actually working pretty well. Want to share here.

53. 12:45

    In terms of the core skills, uh, we usually prioritize the stronger software engineer skills over the AI expertise. This might be controversial, but, uh, uh, uh, we can discuss if you are interested.

54. 13:01

    Second is, instead of hiring for experience or degrees, we hire for the potential because this field is evolving so fast, most of the experience might be outdated.

55. 13:14

    In case you won't be able to find a single engineer with all the qualifications we're showing here, uh, the way we are solving this problem is to hire or diversify the team.

56. 13:26

    So, so for example, uh, in our team, we have some full stack software engineers. We have, uh, data scientists. We have AI engineers and, uh, data engineers. We also have, uh, fresh grads, uh, from the top research university and also, uh, some people from the startup background.

57. 13:49

    And then we put them together, uh, into the project. What we're seeing is, based on those collaboration, those strong engineers start to pick up new skills in the project, and very soon, they started to grow into these ideal candidates.

58. 14:04

    Uh, lastly is, uh, want to emphasize is, uh, the critical thinking. Uh, one constant topic, uh, in our team meeting is, uh, no matter what we're building right now, it will be outdated within a year or even less than six month.

59. 14:20

    So we consistently evaluate the latest open source package, talking with vendors, and deprecate our solution more proactively.

60. 14:30

    Cool. Let's talk about the sa-- uh, some, uh, key takeaways, uh, especially on the tech stack choice. If possible, we strongly recommend Python. We started with Java and Python.

61. 14:43

    Uh, there are some back and forth of the debate internally, but finally, we picked Python, and I think that's the right choice, especially most research and open source, uh, are in this space.

62. 14:55

    Based on our experience, it's also scalable. In terms of the, uh, key components you want to build in this platform, the first one is a prompt source of truth.

63. 15:06

    Prompt in some way is like a traditional model parameters. You want to have a really robust systems to version control your prompt. This is really, really critical for the operational stability.

64. 15:18

    You don't want accidentally added to your prompt in production and, uh, see some really side effect.

65. 15:25

    Second key component is on the memory. I think in today's, uh, meeting, uh, I-I-I mean, today's talk, someone already talked about it. Memory is a really key component to inject your rich data into the agent experience.

66. 15:40

    Lastly, in the agent era, uh, one key new component we are building is on the uplifting our APIs into skills which can be called from the agent easily. So you can, uh, build some surrounding tooling and infrastructure to support this need.

67. 16:02

    All right. Let's talk about how to, uh, scale this solution and got-- guide it, adopt it. Uh, from our experience, instead of trying to build this full-fledged platform at the beginning, try to solve immediate need.

68. 16:16

    For example, we started with a simple Python library to support orchestration. Then we started to grow into all the components we're seeing here. Second is, uh, focus on the infrastructure and the scalable solution.

69. 16:31

    At LinkedIn, we actually have a pretty good success story by leveraging our, uh, messaging infrastructure, uh, to be as a memory layer. Uh, it's both cost efficient and scalable.

70. 16:44

    Lastly is, uh, focus on the developer experience. By the end of the day, this platform is trying to help developer to be as productive as possible. Their adoption is a key for the success.

71. 16:57

    If you can design this platform, please focus on, uh, how to align your technology with their existing, uh, workflow, so it will ease adoption and, uh, be more successful.

72. 17:11

    Uh, we actually have lots of low-level details on the technical side. Uh, if you are interested, please check out our engineering blog post, uh, on LinkedIn, uh, by Karthik, Sandeep, and myself.

73. 17:24

    Uh, with that, uh, thank you for your attention, and, uh, if you are having more questions, happy to answer that after the talk. Thank you. [upbeat music]