Building Context-Aware Reasoning Applications with LangChain and LangSmith

[on-hold music] Thank you guys for having me, and then thank you guys for being here.

This is, this is maybe one of the most famous, uh, screens of, of twenty twenty-three. And, and yet I believe, and I think we all believe, and that's why we're all here, that this is just the beginning of a lot of amazing things that we're all going to create.

Because as good as ChatGPT is and as good as the language models that underlie them are, by themselves, they're just the start. By themselves, they don't know about current events, they cannot run the code that you write, and they don't remember their previous interactions with you.

In order to get to a future where we have truly personalized and actually helpful AI assistants, we're gonna need to take these language models and use them as one part of a larger system.

And that, and that's what I think a lot of us in here are, are trying to do. These systems will be able to produce seemingly, you know, amazing and magical experiences.

They'll understand the appropriate context, and they'll be able to reason about it and respond appropriately.

At LangChain, we're trying to help teams close that gap between these magical experiences and, and, and the work that's actually required to get there, and we believe that behind all of these seemingly magical product moments, there is an extraordinary feat of engineering.

And that's why it's awesome to be here at the AI Engineering Summit. And so I'm gonna talk a little bit about some of the approaches that we see work for developers when they're building these context-aware reasoning applications that are gonna power the future.

So first, I'm gonna talk about context. A-and when I say context, I mean bringing relevant context to the language model so that it can reason about what to do.

And bringing that context is really, really important because if you don't provide that context, no matter how good the language model is, it's not gonna be able to, to figure out what to do.

And, and then, so the first type of context, and probably the most common type of context that we see people bringing to the language model, we see them bringing through this instruction prompting type of approach, where they basically tell the language model how to respond to specific scenarios or specific inputs.

Um, this, this is pretty straightforward, and I think the way to think about it is if, uh, you have a new employee who shows up on the first day of work, you give them an employee handbook, and it tells them how they should behave in certain scenarios.

And, and I equivalate that to kind of like this instruction prompting technique. It's, um, you know, it's pretty straightforward. I think that's why people start with it. And as the models get better and better, this zero-shot type of prompting is gonna be able to carry a lot of the relevant context for how you expect the language model

to behave. There's some cases where telling the language model is actually quite hard, and it becomes better to give it some few-shot examples. It becomes better to give it examples where you show the language model how to behave rather than just tell it how to behave.

And so I think a few concrete places where this works is where it's actually a little bit difficult to describe how exactly the language model should respond. So tone, I think, is a good use case for this, and then also structured output is a good use case for this.

You can give examples of the, of the structured output format. You can give examples of the output tone a little bit more easily than you could describe in language my particular tone.

The structured output is a little bit-- You can describe structured output, but I think as it starts to get more and more complicated, giving these really specific examples can help.

The next type of context is maybe the most, um-- You know, it pops to the mind most when you hear of context. I mean, you hear about bringing context to the language model.

Contrasting this with the first two, retrieval-augmented generation uses context not to decide, uh, how to respond, but to decide kind of like what to base its response in. So the kind of like canonical thing is you have a user question, you do some retrieval strategy, you get back some context, you pass that to the language model, and

you say, "Answer this question based on the context that's provided to you." Um, and so this is a little bit different from the instructions. It's, it's maybe the same as asking someone to take a test with, uh, like an open-book test.

You can look at the book. You can look at the answers. And in this case, the answers are the text that you pass in to this context.

And, and then the fourth way that we see people s- providing context to language models is through fine-tuning, so updating the actual weights of the language model. Um, th-this is still kind of like in its infancy, and I think we're starting to figure out how best to do this and what scenarios this is, this is good to

do in. Um, s- One of the things that we've seen is that this is good for the same use cases where few-shot examples are kind of good. It takes it to another extreme.

Um, and so for tone and structured data, uh, parsing, these are two use cases where we've seen it pretty beneficial to start doing some fine-tuning. And, and the idea here is that, yeah, it can be helpful to have three examples of how your model should respond and what the tone there should be.

But what if you could give it ten thousand examples and it updates its weight accordingly? And so I think for those where, where the output is in a, a, a specific format, and again, you need more examples, you need to show it a lot more than you can tell it, this is where we see fine-tuning starting to

become helpful. And I think we'll see that grow more and more over time. So we've talked about context. Um, and now I wanna talk a little bit about the reasoning bit.

And I think this is the most exciting and the most new bit of it as well. And so we've tried to think and categorize some of the approaches that we've seen to allow, uh, these applications to do this reasoning component.

And, and so we've listed a few of them out here and tried to, tried to discern a few different axes along which they kind of vary. So if we think about kind of like just plain old code, this is kind of like the way things were, you know, in the-- [chuckles] like a year ago, so a long, long

time ago. Um, and so in code, you kind of like-- You, you-- It's all there. It's declared if it says what to run. Um, it says what the outputs are, what, what steps to take, things like that.

We start adding in a language model call, and so this is like the, the simplest form of, of these reasoning applications. And here you're using the language model to determine what the output should be, but that's it.

You're not, you're not using it to take actions yet, nothing fancy. You're just using it to determine what the output should be, and it's just a single language model call.

So you're providing the context and, and then you're, uh, returning the output to the user.

If we take it up a little bit, then we start to get inter-- like chain of, of language model calls or a chain of language model call to API back to language model.

And so this can be u- this is again used to, uh, decide the, the, the steps of the output. Um, a-and here there's, uh, multiple calls that are happening, and this can be used to break down, um, more complex task into individual components.

It can be used to insert knowledge dynamically in the middle of kind of like one language model call. Then you go fetch some knowledge based on that language model call, and then you do another one.

Um, but importantly here, the steps are known. You do this, and then you do this, and then you do this. And so it's a chain of events. And that starts to change a little bit when you use a router.

Um, and, and so in here, you're now using the language model call to start determining which steps to take. So that's the big difference here. It's no longer just determining the output of the system, but it's determining which steps to take.

And so you can use it to determine, uh, which prompts to use, so route between a prompt that's really good at math problems versus a prompt that's really good at, at r- uh, writing English essays.

You can use it to route between, uh, language models. So one model might be better than another. You might wanna use Claude because of its long context window, or you might wanna use GPT-4 because it's really good at reasoning.

And so having the language model look at the question and decide whether it needs to reason or whether it wants to respond in a long-form fashion, you can determine which branches to go down.

Or I think m-- uh, another common use case is using it to determine which of several tools to take. So do I wanna call this tool, or do I wanna call this tool?

And what sh-- what should the input to that tools be? And so we have this router here. And I think, um, before going on to the next step, the, the main thing here that distinguishes it from that step is that there's no kind of like cycles.

You don't kind of get these, these loops. Um, uh, you're just choosing kind of like which branch to go down. Once you start adding in these loops, this is where we see, uh, a lot more complex applications.

Um, this is, this is-- these are things that we often see being called agents kind of like out, out in the wild, and it's essentially kind of like a while loop.

Um, and then in that loop, you're doing a series of steps, um, and the language model is determining which steps to do. And then at some point, there's a point where it can choose whether to end the loop or not.

Um, and if it ends the loop, then you finish and return to the user. Otherwise, you go back and, and, and continue the loop. And so here you get the language model deciding what the outputs are.

It decides what steps to take, and you do have these, these cycles. Um, the last thing, um, and I think the, the-- t-this is, this is largely what we would describe as kind of like what Auto-GPT did that took the world by storm, is this idea of an agent, um, where you kind of like remove a lot

of the, um, the, the kind of like guardrails around what steps to take. So here, the sequences of steps that are available are, are almost like determined by the LLM.

And what I mean by this is that here's where you can start doing things like adding in tools that the, that the language model can take. So if you guys are familiar with the Voyager paper, it, it starts adding in tools and building up a skill set of tools over time.

And so some of the actions that the language model can take are, are dynamically created. Um, and then I think the other big thing here is that you remove some of the scaffolding from the state machines.

Um, so s-some of the, uh-- If I go back a little bit. So a lot of the-- these kind of like cycles that we see in the wild break things down into discrete states.

The most common one that we see are kind of like plan, execute, and validate. So you ask the language model to plan what to do, it then goes do it, and then you validate it often with a language model call or something like that.

And I think the big difference between that and then the autonomous agent style thing is that here you're implicitly asking the agent to do all of those things in one go.

It should know when it should plan, it should know when it should, uh, validate, and it should know when it should kind of like determine what action to take.

And you're asking it all to do that implicitly. You don't have these kind of like distinct sequences of steps laid out in the code.

And so this is a little bit about how we're thinking about it. I think the, the thing to, uh, th-the thing that I like to say when saying this as well, which goes back to the beginning, is that the main thing that we think is it's still just extremely early on in the space.

We still think it's the beginning. And this could, you know, in, in three months be kind of irrelevant as the space progresses. So I would just keep that in mind.

If we think about kind of like some of the magical experiences like this where it can reason over the relevant context, what is it gonna take to kind of like build it under the hood?

What is the engineering that's gonna go into all these seemingly magical experiences? And so this is an example of, of what could be going under the hood of something like this.

It's gonna be a challenging experience to build these complex systems, and that's why we're building some of the tooling like, like this, what you see here, to help debug, understand, and iterate on these systems of the future.

And so what exactly are the challenges associated with building these complex context-aware reasoning applications? The first is kind of just the orchestration layer. So, uh, figuring out which of the different reasoning, kind of like cognitive architectures you should be using.

Should you be using, uh, a simple chain? Should you be using a router, a more complex agent? And I think the thing to remember here is that it's not necessarily that one is better than the other or superior to the other.

They all have kind of like their pros and cons and strengths and weaknesses. So chains are really good because you have more control over the sequence of steps that are taken.

Agents are better 'cause they can more dynamically react to unexpected inputs and handle edge cases. And so being able to choose the, the right cognitive architecture that you want and being able to quickly experiment with a bunch of other ones are part of what inspired the initial release of LangChain and, and, and kind of how we aim

to help people, uh, prototype these, these types of applications. And then LangSmith, which is this thing here, provides a lot of visibility into what is actually is going on as these, uh, as these applications start to get more and more complex, understanding what exact sequences of, of tools are being used, what, what exact sequences of language model

calls are being made becomes increasingly important. Another big thing that we see people struggling with and spending a lot of time on is good old-fashioned data engineering. So a lot of this comes down to providing the right context to language models, and the right context is often data.

And so you need to have ways to load that data, you need to have ways to transform that data, tr- transport that data, and then you often wanna have observability into what exact data is getting passed around and where.

And so LangChain itself has a lot of open source kind of, like, modules for loading that data and transforming that data. And then LangSmith we often see being really useful for debugging what exactly does that data look like by the time it's getting to the language model.

Have you extracted the right documents from your vector store? Have you transformed them and formatted in the right way where it's clear to the language model what's actually in them?

These are all things that you're gonna wanna be able to, to debug so there's no little small errors or small issues that pop up.

And then the, the third thing that we see a lot of people spending time on when building these applications is just good old-fashioned prompt engineering. Um, so the main new thing here is language models, and the main way of interacting with language models is through prompts.

And so being able to understand what exactly does the fully formatted prompt look like by the time it's going into the language model is, is really important. What d- like, how are you combining, uh, the system instructions with maybe the few-shot examples, um, any retrieved context, the chat history that you've got going on, any previous steps that

the agent took? What does this all look by the time it gets to the language model? And w- and, and what does this look like in the middle of this complex application?

So it's easy enough to, to kind of, like, test and debug this if it's the first call, the first part of the system, but after it's already done three of these steps, if you wanna kind of, like, debug what that prompt looks like, what that fully formatted prompt looks like, being able to do that, um, it becomes

increasingly difficult as the systems kind of, like, scale up in, in their entangledness. And so we've tried to make it really easy to, to hop into any kind of, like, particular language model call at any point in time, open it up in a playground like this so you can edit it directly a- a- and experiment with that

prompt engineering and, and, and go kind of like, uh, change some of the instructions and see how it responds or swap out model providers so that you can see if another model provider does better.

Another big challenge with these language model applications, and it's probably worth a, a talk on its own, is evaluation of them. And so I think evaluation is really hard for a few reasons.

I think the two primary ones are a lack of data and a lack of good metrics. So comparing to traditional kind of, like, data science and machine learning, with those you generally started with a data set.

You needed that to build your model, and so then when it came time to evaluate it, you at least had those data points that you could look at it and evaluate on.

And I think that's a little bit different with a lot of these, um, LLM applications because these, these models are fantastic zero-shot kind of, like, l- learners. That's the-- that's kind of, like, the, the whole exciting bit of them.

And so you can get to a working MVP without building up kind of, like, any data set at all. And that's awesome, but that does make it a little bit of a challenge when it comes to evaluating them because you don't have these data points.

And so one of the things that we often encourage a lot of people to do, and try to help them do as well, is build up these data, these data sets and iterate on those.

And those can come from either labeling data points by hand or looking at production traffic and pulling things in, um, or auto-generating things with, with LLMs. The second, um, big challenge in evaluation is lack of metrics.

Um, I think most traditional kind of, like, quantitative metrics don't perform super well for, for large unstructured outputs. A lot of what we see people doing is still doing a kind of, like, vibe check to kind of, like, see how the model's performing.

Um, and, and as unsatisfying as that, as that is, I still think that's probably the best way to gain, uh, kind of like, um, intuition as to what's going on.

And so a lot of what we try to do is make it really easy to observe the outputs and the inputs of the language models so that you can build up that intuition.

Um, in terms of more quantitative and systematic metrics, we're, we're very bullish on LLM-assisted evaluation, so using LLMs to evaluate the outputs. Um, and th- and then I think maybe the, the biggest thing that we see people doing in production is just keeping track of feedback, um, whether it be direct or indirect feedback.

So do they leave kind of, like, a thumbs up or a thumbs down on your application? That's an example of direct feedback where you're gathering that. An example of indirect feedback might be if they click on a link or that that might be a good thing that you provided a good suggestion.

Or if they, if they respond really confused to your chatbot, that might be a good indication that your, uh, chatbot actually did not perform well. And so tracking these over time and, and doing A/B testing with that using kind of, like, uh, traditional A/B testing software can be, can be pretty impactful for gathering a sense online of

how, of how your model's doing. And then the last interesting thing that we're spending a lot of time thinking about is collaboration. So as these systems get bigger and bigger, they're doubtless gonna be a collaboration among a lot of people.

Um, and so who exactly is, is working on these systems? Um, is it, is it all AI engineers as we're here today? Is it a combination of AI engineers and data engineers and data scientists and, and product managers?

And I think one of the interesting trends that we're seeing is it's still a little bit unclear what the best skill sets for this new AI engineer-type role is.

And, and there could very well be a, a bunch of different skill sets that are valuable. So going back to kind of, like, the two things that we see making up a lot of these applications, the context awareness and the reasoning bit.

The context awareness is bringing the right context to these applications. You often need kind of, like, a data engineering team to get in there and assist with that. The reasoning bit is often done through prompting, and oftentimes that's best done by non-technical people who can really outline kind of, like, the exact specification of the app that they're

building, whether they be product managers or subject matter experts. And so how do you enable collaboration between these two different types of folks, and, and what exactly does that look like?

I don't think that's something that anyone kind of, like, knows or indefinitely hasn't solved, but I think that's a really interesting trend that, that we're thinking a lot about going forward.

And so I think, like, the, the, the main thing that I wanna leave you all with is that the big thing that we believe is that it's still really, really early on in this journey.

It's just the beginning. As crazy as things have been over the past year, they're hopefully gonna get even crazier. You saw an incredible demo of, of GPT-4V. Things like that are gonna change it.

And so we think behind all of these things, it's gonna take a lot of engineering, and we're trying to build some of the tooling to help enable that. And I think you guys are all on the right track towards becoming those types of engineers by being at a conference like this.

So thank you, Swyx, for having me. Thank you guys for being here. Have a good rest of your day. [audience applauding] [upbeat music]