AI Engineer Summit 2025
Lessons from Building LinkedIn's GenAI Platform
Read the talk
From prompt calls to LinkedIn’s GenAI platform
LinkedIn’s path from generated articles to recruiting agents shows how product needs shape orchestration, memory, observability, and the team responsible for them.
From a talk by Xiaofeng Wang
Before you start: Familiarity with LLM prompts, APIs, and basic Python will help; retrieval, memory, and agent orchestration are explained as they arise.
A prompt, an article, and two technology stacks
LinkedIn’s Collaborative Articles began with a simple interaction: generate an article, then invite members to contribute their expertise. Xiaofeng Wang describes this 2023 product as LinkedIn’s first formal GenAI feature and identifies GPT-4 as the model used to produce the long-form content. From the application’s perspective, the mechanism was straightforward: a prompt went in and a string came out.
Supporting that interaction required a gateway to centralize model access and Python notebooks for prompt engineering. But development and serving lived in different worlds: Python on the backend and Java in the online application. These were useful components, not yet what Wang would call a platform. The initial architecture solved access to generation; it did not yet provide a shared way to build richer AI experiences.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Personalization makes shared infrastructure necessary
The limitation became apparent when products needed LinkedIn’s own data. By mid-2023, the next generation of experiences—internally called Copilot or Coach—needed to answer personalized questions. Consider job fit: the application looks at a member’s profile and a job description, then uses retrieval-augmented generation, or RAG, to provide recommendations about whether the member fits the role. A generic article prompt cannot supply that context by itself.
The team built a Python SDK on top of LangChain to orchestrate LLM calls and integrate LinkedIn’s infrastructure. Developers could assemble an application through this shared layer instead of connecting each infrastructure service themselves. This also motivated unifying the technology stack: transferring prompts developed in Python into Java was costly and introduced errors.
Two additional capabilities made the SDK more than a wrapper around model calls. A prompt source of truth versioned prompts and provided structure for meta prompts. Conversational memory tracked LLM interactions and retrieved content so that the application could inject that context into subsequent interactions. Prompt management controlled the instructions; memory supplied the evolving context needed for a conversational product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recruiting agents need execution and discoverable skills
The next product step was LinkedIn Hiring Assistant, which Wang describes as the company’s first real multi-agent system. It launched in charter in 2024, initially for selected customers and LinkedIn recruiters. Wang frames the recruiting workload broadly: posting jobs, evaluating hundreds of candidates, and reaching out to them. The system’s purpose was to automate tedious work across that process, rather than merely return advice in a conversation.
That change expanded the Python SDK into a distributed agent orchestration layer. It had to handle distributed execution, retry logic, and traffic shifting. An agent that chooses and performs actions needs more operational support than an application that makes a single model request.
Actions also require a usable interface to enterprise APIs. The team built tools for developers to publish their APIs into a centralized skill registry. The registry addressed both discovery—finding an available skill—and invocation—calling it from an application. Publishing an API as a skill made the existing service accessible through the agent platform’s common machinery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remember interactions and inspect autonomous behavior
Memory evolved alongside execution. Beyond retaining conversational content, experiential memory extracted, analyzed, and inferred contextual knowledge from interactions between users and agents. The platform organized memory into working, long-term, and collective layers. The distinction is useful: preserving an interaction gives the system a record; extracting contextual knowledge makes that interaction useful as experience.
Autonomy made observability essential. Because an agent can decide which API or LLM to call, its behavior is difficult to predict in advance. LinkedIn built an in-house solution on OpenTelemetry to capture fine-grained telemetry that could be used to replay agent calls. An analytics layer on top of those records then helped guide future optimization. The platform needed visibility into the agent’s execution, not just its final response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One interface to a larger AI ecosystem
These investments formed four main areas of the GenAI application platform:
| Area | Responsibility |
|---|---|
| Orchestration | Coordinate model and agent execution |
| Prompt engineering | Manage instructions and their versions |
| Tools and skills | Support discovery and invocation |
| Content and memory | Supply context across interactions |
The application platform brought these concerns together, but it did not own the entire AI ecosystem.
Sister teams handled model work such as fine-tuning open-source models, responsible AI policies and standards, and the machine-learning infrastructure that hosted models. The GenAI platform’s role was to give application developers a unified interface to that ecosystem. Developers could use those capabilities without first learning every underlying service and integration.
Wang gives model selection as a concrete example: in LinkedIn’s SDK, changing one parameter in one line of code could switch an application from an OpenAI model to an on-premises model. Developers still had to do prompt engineering for the selected model; the abstraction removed infrastructure integration work, not differences in model behavior. Centralization also created a place to enforce best practices and governance, helping developers build applications efficiently and responsibly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The boundary between optimization and serving moves
The platform emerged piece by piece, which raises a reasonable question: with so many vendors offering GenAI tooling, does a company need this platform capability, and should it build or buy it? Wang’s answer addresses the need for the capability rather than selecting a vendor or prescribing that every component be built internally. The reason is a change in how AI applications are developed.
In the traditional arrangement Wang describes, model optimization and serving have a relatively clear boundary. AI engineers and product engineers can use different stacks and need not work in the same codebase. In a GenAI application, that separation fades: engineers on either side can change the overall system’s performance.
| Development concern | Traditional model-centered system | GenAI application |
|---|---|---|
| Optimization and serving | Separate phases | Intertwined work |
| Engineering stacks | Can remain separate | Need shared tooling |
| System improvement | Divided by role | Spans both groups |
Once optimization reaches into application behavior, tooling and engineering practices have to support both groups working on the system.
Wang connects this to Berkeley AI Research’s concept of a compound AI system: a system that tackles AI tasks through interacting components, including multiple model calls, retrievers, or external tools. Improving such a system requires skills that cross the traditional AI-engineering and product-engineering boundary. A shared platform bridges that gap by making the components usable through common interfaces and practices.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the team’s range through collaboration
That platform remit creates a demanding hiring profile. Looking at the strongest engineers on his team, Wang identifies infrastructure integration skills, a developer product-management sensibility for interface design, AI and data-science knowledge, and the ability to learn new techniques while remaining hands-on. Finding all of these in one person is rare—rare enough that he jokes the candidate would be worth more than a unicorn. Hiring therefore requires deliberate trade-offs.
- Core engineering skills: Prioritize strong software engineering over existing AI expertise.
- Potential: Hire for the ability to learn rather than relying mainly on experience or degrees, because the field changes quickly.
- Complementary backgrounds: Build the missing range across the team. LinkedIn’s mix included full-stack engineers, data scientists, AI engineers, data engineers, recent research-university graduates, and people with startup experience.
The development mechanism was joint project work. Wang reports that strong engineers acquired complementary skills through collaboration and grew toward the broader ideal profile.
Critical thinking matters after the team is hired, too. Wang describes a recurring team premise: what they are building may become outdated within a year, or even in less than six months. That premise drives an ongoing practice of evaluating open-source packages, talking with vendors, and proactively deprecating internal solutions. Building the platform does not require defending every component indefinitely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose Python and treat prompts as production parameters
Returning to the technology stack, Wang recommends Python where possible. LinkedIn began with Java and Python, debated the choice internally, and ultimately chose Python because of its alignment with research and the open-source ecosystem. Wang also reports that Python proved scalable in LinkedIn’s experience; he supplies no workload measurements for a language comparison.
Prompts deserve the operational discipline applied to model parameters. A prompt source of truth needs robust version control because changing instructions changes application behavior. An accidental production edit can create unwanted side effects even when the surrounding application code remains unchanged.
A small Python example makes the distinction between authoring a prompt and selecting a production version concrete. Here, the application explicitly selects job-fit-v1; adding job-fit-v2 to the catalog does not change that selection:
python
from types import MappingProxyType
PROMPTS = MappingProxyType({
"job-fit-v1": (
"Assess this member's fit for the job using the supplied facts.\n"
"Profile: {profile}\nJob description: {job_description}"
),
"job-fit-v2": (
"Compare the supplied profile with the job requirements.\n"
"Separate supported matches from missing information.\n"
"Profile: {profile}\nJob description: {job_description}"
),
})
PRODUCTION_PROMPT_VERSION = "job-fit-v1"
def render_job_fit_prompt(profile: str, job_description: str) -> str:
template = PROMPTS[PRODUCTION_PROMPT_VERSION]
return template.format(
profile=profile,
job_description=job_description,
)
This illustrates explicit version selection, not LinkedIn’s internal SDK. The production safeguard depends on preserving released versions and making changes to the selected version deliberate.
The other essential investments follow the product’s needs. Memory injects rich enterprise data into agent experiences. Tooling around existing APIs turns them into skills that agents can call easily. Together, these components give applications controlled instructions, relevant context, and access to actions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Grow from immediate needs into developers’ workflows
The route to adoption starts smaller than the finished architecture suggests. Solve an immediate need instead of building a full platform at the outset. LinkedIn began with a simple Python orchestration library and expanded into the broader set of components as product requirements grew. The useful unit of progress was a capability developers could use, not completion of the entire platform diagram.
Scaling also meant reusing infrastructure. LinkedIn used its existing messaging infrastructure as a memory layer, an approach Wang describes as cost efficient and scalable. The lesson is to look for infrastructure that already provides the foundations a new capability needs, rather than assuming that an AI-specific requirement demands an entirely new storage system.
Developer experience determines whether those capabilities are adopted. The platform exists to make developers productive, so its technology needs to fit their existing workflows. A technically capable system that makes everyday development harder misses the reason for building it.
For the lower-level implementation details, Wang points to the team’s engineering article on LinkedIn’s GenAI application stack, by Karthik R., Xiaofeng Wang, and Sandeep Jha.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The team's technical account of Python adoption, LangChain integration, prompt management, skill discovery, memory infrastructure and model access.
The public framework repository underlying LinkedIn's historical Python SDK; its current code has evolved since the talk.
Current documentation for the telemetry project used as the foundation of LinkedIn's in-house observability solution.
Further reading
The October 2024 charter launch announcement describes recruiting tasks, human oversight and planned capabilities.
LinkedIn explains how AI-assisted article starters connect with contributions from relevant member experts.
Read the complete timestamped transcript
- 0:00
[on-hold music] Uh, it's my pleasure here to share our journey on building out LinkedIn's GenAI platform.
- 0:23
Uh, my name is Xiaofeng Wang, ad manager of GenAI Foundation.
- 0:29
Uh, let me try it one more time. Cool. Uh, in today's talk, I'd like to first share our journey on building out this platform, especially on why we're building it, how we build it, and what we're building it.
- 0:44
After that, we will talk about, uh, some thought process on why this platform is critical for today's agent world. Um, hopefully after that, you agree with me this is critical component in your component-- uh, uh, in your company, and you also want to build this team.
- 1:01
I want to share some tips on how to build such a team, how to hire for such a team. Towards the end, we will share some key takeaways and lessons learned.
- 1:11
Before we dive into this application, uh, platform journey, I think it's important to first talk about the GenAI product experience because that's essentially what our platform is supporting for.
- 1:26
Back in twenty-twenty three, LinkedIn launched the first, uh, formal GenAI feature called Collaborative Articles. This is a kind straightforward, uh, GenAI feature if we are thinking in today's standard because it's a very simple prompt in, string out type of application.
- 1:46
Uh, we leverage ChatGPT-- Uh, I mean, we leverage GPT-4 model, uh, to create the long content, uh, articles on the platform and then invite our members to comment on it.
- 1:59
At this stage, our team helped to build some key component behind the scene, including the gateway to centralize the access to the model, uh, some Python notebook for the prompt engineering.
- 2:11
Uh, but at this time, we actually have, uh, two different tech stacks, uh, to serve the, uh, experience. In the online phase, we use Java, and in the back end, we use Python.
- 2:23
Uh, we wouldn't call this as a platform at this time.
- 2:28
Very soon we realize, uh, there are some limitation for this simple approach, especially it lacks the capability to inject our rich data into the product experience. Then in the mid-2023, we started to develop the second generation of the GenAI product.
- 2:47
Uh, internally, we call it Copilot or Coach. Here we're showing one popular such experience on LinkedIn right now. Uh, basically it looks at, uh, your profile and the job description and then, uh, use, uh, some RAG process to give you personalized recommendation on if you are a good fit to the job.
- 3:10
At this time, we started to build, uh, some platform capability. Uh, specifically in the center of our platform, we build the, uh, Python SDK on top of the popular, uh, LangChain framework to orchestrate LLM calls.
- 3:27
And it also provide the key value to integrate with our large-scale infrastructure, uh, in this SDK, so our developers can easily assemble an application. We started to unify the tech stack at this stage because we realized it's really costly to transfer the Python prompt into the Java world, not to mention the error during this process.
- 3:51
We started to invest on the prompt management or prompt source of truth. This is a sub-module at this stage, uh, to help developers to version their prompt and to provide some structure around their meta prompt.
- 4:07
Uh, the most important piece I'd like to call out here is conversational memory. Uh, this is, uh, infrastructure to help to keep track of the LLM interactions and retrieval content and then inject those content into the final product.
- 4:24
It will help us to build this kind of conversational, uh, bot.
- 4:31
Now, uh, zooming to this year, uh, actually in the last year, uh, we launched our first ever, uh, real multi-agent system called, uh, LinkedIn Hiring Assistant. Uh, this is, uh, multi-agent systems to help our recruiters to do their work, uh, efficiently, especially it automate several tedious task, uh, normally recruiter need to do manually, like,
- 4:55
um, uh, post the job, um, and evaluate hundreds of candidates, then, uh, reach out to them.
- 5:06
Our platform also start to evolve into the agent platform. Uh, from the framework side, we extend the support of the Python SDK into a more large-scaled, uh, distributed agent orchestration layer.
- 5:21
It will handle the distributed agent execution and also handle the more complicated scenarios like retry logic and the traffic shift. Uh, for folks who build agent, uh, I think you probably know the skills or APIs are one key aspect of the agent because we expect, uh, this agent to perform some action.
- 5:45
One investment we did at this, uh, time is around the skill registry. Basically, we have a set of tools, uh, to help our developers to publish their API into this centralized skill registry.
- 5:59
This skill registry can handle the skill discovery problem-
- 6:03
Skill invocation problem. So in your application, it's actually very easy to call the API to perform some task.
- 6:11
Another key component, uh, we invest at this stage is on the memory. In addition to the conversational memory, we extend, uh, its capability into the experiential memory. Essentially, it's a memory storage to, uh, extract and analyze, and infer the contextual knowledge from the interaction between the agent and our user.
- 6:33
We also organize this memory into different layers, including the, um, uh,
- 6:40
working memory, long-term memory, collective memories. Uh, this can help our agent to be aware of the surrounding content. Uh, lastly, at this, uh, time, we also realize the operability is super important because agent, uh, one key aspect to define agent is autonomous, right?
- 6:59
Uh, because agent can decide what API they can call, uh, what LL, uh, uh, LLM they need to call. So it's actually very hard to predict its behavior. So we started to invest on the operability.
- 7:13
Uh, particularly, we build our in-house solution on top of the OTEL to keep track very low level granularity of the, uh, telemetry data. So we can use this data to replay the, uh, agent call.
- 7:27
And we also add a actual layer of the analytics on top of it. So we can use that to guide the future optimization of our agent systems.
- 7:37
Let's put together all the components we build for this platform. Uh, we can classify them into four layers, basically, including the orchestration, prompt engineering, tools and the skills invocation, content, and the memory, uh, management.
- 7:54
Uh, of course, that's not everything in the LinkedIn GenAI ecosystem. Uh, in addition, we have our sister teams to build out the modeling layer, like fine-tune the open source models, responsible AI layers to make sure the agent is behave according to our policy and standard, and also the, uh, AI platform or machine learning infrastructure team to host
- 8:19
those models. The key value proposition for this, uh, GenAI platform is actually to, uh, be the unified interface for this complex ecosystem.
- 8:34
So our developers don't need to necessarily understand all those individual box when they build, uh, their application. Instead, they can leverage our platform to quickly access to this entire ecosystem.
- 8:47
Uh, for example, uh, in our SDK, the developer can just switch one parameter in the one line of the code to switch from the open AI model to our on-prem model.
- 9:00
Of course, they still need to do the prompt engineering, but that reduce a lot of the complexity on the infrastructure integration phase.
- 9:09
Uh, last but the most important is because of this is a centralized platform, uh, it provide a place to enforce the best practice and governance. So we can make sure our developers are building the applications efficiently but also responsibly.
- 9:28
As you can see from our journey, we actually started to build this, uh, platform piece by piece, and then this platform start to emerge. If we take one step back and think, oh, do we really need this platform at this time?
- 9:41
Especially there are lots of, uh, uh, uh, vendor product, uh, on this space. Shall we buy it, build it? And why do we need to buy it or build it?
- 9:51
Uh, here are some thoughts. Um, the short answer is yes. The reason behind it is, uh, we feel like, uh, GenAI is a totally different and new AI systems compared to the traditional AI systems.
- 10:04
So in the traditional AI systems, there's a clear cutoff between the, uh, AI model optimization phase and the model serving phase. So AI engineers and the product engineers can operate in two different tech stack.
- 10:19
Uh, they usually don't, uh, need to, uh, work on the same code base. But in the GenAI systems, what we're seeing is this line between the optimization phase and the serving phase disappear.
- 10:33
Basically, everyone is AI engineer who can optimize the overall system performance. This actually create the new challenge of the tooling and the best practice in the company.
- 10:47
Essentially, we think these GenAI systems or, uh, agent systems is a compound AI system. Here, we borrow the definition from Berkeley AI Research Lab. A compound AI system can be defined as a system which tackles AI tasks using multiple interacting components, including multiple calls to model retrievers or external tools.
- 11:10
As you can see, this is actually skill across AI engineer and product engineer. And I believe this, uh, GenAI platform is trying to bridge this gap.
- 11:23
To summarize, uh, we believe this platform is critical for your success, mainly because it can bridge the skill gaps between those two group of engineers.
- 11:34
Okay. Let's say if you want to build this, uh, platform in your company and how to hire it is a frequent question, uh, I heard. Um, I basically look into, uh, my great engineering team and, uh, extract all the key qualifiers from those top engineers.
- 11:54
And, uh, I put all the qualifications here. Uh, the ideal candidate in this team is a strong software engineer, uh, who can build infrastructure integration. They have a good developer, uh, PM skills to design the interface.
- 12:12
Uh, ideally, they have the AI and the data science background to understand the latest techniques. They are the people who can learn from the latest techniques, but at the same time, they are hands-on.
- 12:26
Unfortunately, [chuckles] it's really hard to get those candidates. If you get them, uh, it's probably worth more than a unicorn. [laughs]
- 12:33
Realistically, we are making multiple trade-offs in the hiring. Uh, here are some principle, uh, we follow, and it's actually working pretty well. Want to share here.
- 12:45
In terms of the core skills, uh, we usually prioritize the stronger software engineer skills over the AI expertise. This might be controversial, but, uh, uh, uh, we can discuss if you are interested.
- 13:01
Second is, instead of hiring for experience or degrees, we hire for the potential because this field is evolving so fast, most of the experience might be outdated.
- 13:14
In case you won't be able to find a single engineer with all the qualifications we're showing here, uh, the way we are solving this problem is to hire or diversify the team.
- 13:26
So, so for example, uh, in our team, we have some full stack software engineers. We have, uh, data scientists. We have AI engineers and, uh, data engineers. We also have, uh, fresh grads, uh, from the top research university and also, uh, some people from the startup background.
- 13:49
And then we put them together, uh, into the project. What we're seeing is, based on those collaboration, those strong engineers start to pick up new skills in the project, and very soon, they started to grow into these ideal candidates.
- 14:04
Uh, lastly is, uh, want to emphasize is, uh, the critical thinking. Uh, one constant topic, uh, in our team meeting is, uh, no matter what we're building right now, it will be outdated within a year or even less than six month.
- 14:20
So we consistently evaluate the latest open source package, talking with vendors, and deprecate our solution more proactively.
- 14:30
Cool. Let's talk about the sa-- uh, some, uh, key takeaways, uh, especially on the tech stack choice. If possible, we strongly recommend Python. We started with Java and Python.
- 14:43
Uh, there are some back and forth of the debate internally, but finally, we picked Python, and I think that's the right choice, especially most research and open source, uh, are in this space.
- 14:55
Based on our experience, it's also scalable. In terms of the, uh, key components you want to build in this platform, the first one is a prompt source of truth.
- 15:06
Prompt in some way is like a traditional model parameters. You want to have a really robust systems to version control your prompt. This is really, really critical for the operational stability.
- 15:18
You don't want accidentally added to your prompt in production and, uh, see some really side effect.
- 15:25
Second key component is on the memory. I think in today's, uh, meeting, uh, I-I-I mean, today's talk, someone already talked about it. Memory is a really key component to inject your rich data into the agent experience.
- 15:40
Lastly, in the agent era, uh, one key new component we are building is on the uplifting our APIs into skills which can be called from the agent easily. So you can, uh, build some surrounding tooling and infrastructure to support this need.
- 16:02
All right. Let's talk about how to, uh, scale this solution and got-- guide it, adopt it. Uh, from our experience, instead of trying to build this full-fledged platform at the beginning, try to solve immediate need.
- 16:16
For example, we started with a simple Python library to support orchestration. Then we started to grow into all the components we're seeing here. Second is, uh, focus on the infrastructure and the scalable solution.
- 16:31
At LinkedIn, we actually have a pretty good success story by leveraging our, uh, messaging infrastructure, uh, to be as a memory layer. Uh, it's both cost efficient and scalable.
- 16:44
Lastly is, uh, focus on the developer experience. By the end of the day, this platform is trying to help developer to be as productive as possible. Their adoption is a key for the success.
- 16:57
If you can design this platform, please focus on, uh, how to align your technology with their existing, uh, workflow, so it will ease adoption and, uh, be more successful.
- 17:11
Uh, we actually have lots of low-level details on the technical side. Uh, if you are interested, please check out our engineering blog post, uh, on LinkedIn, uh, by Karthik, Sandeep, and myself.
- 17:24
Uh, with that, uh, thank you for your attention, and, uh, if you are having more questions, happy to answer that after the talk. Thank you. [upbeat music]