# Building Context-Aware Reasoning Applications with LangChain and LangSmith

Harrison Chase, LangChain

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/cwjs1WAG9CM-building-context-aware-reasoning-applications) · [Video](https://www.youtube.com/watch?v=cwjs1WAG9CM)

## Building Context-Aware Reasoning Applications with LangChain and LangSmith

A useful assistant needs more than a model: it needs relevant context, an appropriate execution structure, and tools for inspecting, evaluating, and improving the whole system.

### A weather question exposes the missing system

[0:16](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=16s)

Ask ChatGPT about the current weather in San Francisco, and the opening example returns an explanation that it cannot access real-time weather data. The question is simple; the missing capability sits outside the model. A standalone language model does not inherently fetch current information, execute the code it generates, or retain previous interactions. This distinction matters in the talk’s 2023 setting: [ChatGPT plugins](https://openai.com/index/chatgpt-plugins/) had already introduced browsing and code execution as surrounding product capabilities. The limitation concerns what the model can do by itself.

![ChatGPT conversation asking about San Francisco weather, with a response explaining that it cannot access real-time weather data.](https://ai.engineer/api/talk-essay-media/0145fe02e2092834ee9d9cb61bcc018e366bcc186700e06c060300ba0484dd86)

ChatGPT cannot answer a question about current San Francisco weather.

**Useful assistants require language models to operate inside a larger system.** That system must supply the relevant context, enable reasoning about it, and support an appropriate response. A seemingly magical product experience therefore depends on substantial engineering underneath. Closing that gap between the experience and its implementation is the goal Harrison Chase sets for LangChain.

### Tell the model, show it, and give it evidence

[1:49](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=109s)

Context is the information a model needs to decide what to do. Greater model capability cannot compensate for information that the application never supplies. The first form is **instruction prompting**: tell the model how to respond to particular inputs or situations. Think of an employee handbook handed to someone on their first day. It describes expected behavior before the employee encounters each scenario. Chase expects stronger models to make these zero-shot instructions increasingly effective.

When behavior is difficult to describe, **few-shot examples** show it directly. A particular tone may be easier to demonstrate than to capture in a verbal specification. Structured output offers another case: a simple format can be described, but increasingly complicated formats benefit from concrete examples of valid responses. Instructions and demonstrations both communicate how the model should behave; examples are especially useful where the description becomes cumbersome.

**Retrieval-augmented generation** supplies a different kind of context: the material on which the answer should be based. Its basic sequence is:

1. Receive the user’s question.
2. Use a retrieval strategy to find relevant material.
3. Pass that material and the question to the model.
4. Instruct the model to answer from the supplied context.

This resembles an open-book test. Behavioral instructions establish how to answer; the retrieved text provides the reference material for answering.

**Fine-tuning** moves learning from examples into updates to the model’s weights. In Chase’s account, this was an emerging practice with useful overlap with few-shot prompting, particularly for tone and structured-data parsing. His contrast between three examples in a prompt and ten thousand training examples illustrates the difference in how many demonstrations can shape behavior; it is not an experiment or a recommended training-set size. When a specific output behavior is easier to show repeatedly than to describe, fine-tuning offers another way to teach it.

### From generating an output to choosing a step

[5:17](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=317s)

Once the model has context, the next design question is how much control it should have over execution. Ordinary code specifies the steps to run. Adding a single language-model call changes who determines the output: the application supplies context, the model generates a response, and the application returns it to the user. The model does not yet select actions.

A **chain** extends this into a known sequence of calls. It might connect several model calls, or run a model call, an API call, and another model call. This allows a complex task to be divided into smaller components. It also allows knowledge to enter partway through execution: an earlier model output can inform what the application retrieves before a later model call. The information changes dynamically, but the sequence remains prescribed in advance.

A **router** gives the model a new responsibility: choosing which step to take. The available branches can differ in several ways:

- **Prompt selection:** Choose a prompt suited to a math problem or one suited to writing an English essay.
- **Model selection:** Choose a model according to the question’s demands. Chase uses **[Claude](https://www.anthropic.com/news/100k-context-windows)** for long context and **GPT-4** for reasoning as contemporary examples, not as a measured comparison between specified model versions.
- **Tool selection:** Choose which tool to call and generate its input.

The defining boundary here is that routing has no cycle. The model selects a branch, but it does not repeatedly return to the same decision point.

### Loops add repeated decisions; autonomy removes scaffolding

[8:01](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=481s)

Adding a loop produces a pattern commonly called an **agent**. Conceptually, it is a `while` loop in which the model chooses steps and decides whether to continue. Ending the loop finishes the task and returns a result to the user; continuing sends execution around again. The model now determines both outputs and successive actions.

The more autonomous style Chase associates with the original [Auto-GPT](https://github.com/significant-gravitas/autogpt) removes further constraints on the available action sequences. One possibility is to expand the available tools during execution. [Voyager](https://arxiv.org/abs/2305.16291v1), his example, accumulates reusable executable skills over time. Those skills enlarge what the agent can do; accumulating them is not a model-weight update.

Another change concerns the structure around the loop. A scaffolded system can explicitly separate **plan, execute, and validate**: ask the model for a plan, carry it out, then check the result, potentially with another model call. A less scaffolded agent must decide for itself when to plan, when to act, and when to validate. Those transitions become implicit model responsibilities instead of distinct sequences encoded in the application.

The architecture table brings these distinctions together: control shifts from code-defined outputs and sequences toward model-selected actions and, eventually, model-influenced action possibilities. Its labels distinguish a scaffolded state machine from the less constrained agent style, even though agent is also commonly used for the broader looping pattern. Chase treats this taxonomy as provisional in a rapidly changing field, rather than a permanent classification.

![Six-row table comparing Code, LLM Call, Chain, Router, State Machine, and Agent across output decisions, step selection, and available step sequences, using code and LLM icons.](https://ai.engineer/api/talk-essay-media/f7764091094ef59a5c590fdf3dee954d2685e4539d1cdfdff226ca0e2cfbe4a9)

Code and LLM calls begin a progression toward greater model control.

### Choose the architecture, then inspect what actually ran

[10:21](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=621s)

The weather assistant becomes an engineering problem when its internal execution is exposed. The displayed debugging interface contains an expanded `AgentExecutor` trace, a `get_weather` tool call, the weather-question input, a completed response, and a Success status. Seeing both the execution tree and the result makes the hidden work behind an assistant response inspectable.

![Dark debugging interface with an expanded execution tree including get_weather, a weather-question input, a completed response, and a Success status.](https://ai.engineer/api/talk-essay-media/4b29a32ffce1e250bcd98aeae379f875303e0593348a878f8a633ca323d9c0c3)

An AgentExecutor trace shows the weather tool call and resulting response.

The first challenge is **orchestration**: selecting the appropriate structure for the application. More model control is not automatically better.

| Structure | Useful property |
| --- | --- |
| Chain | Control over the sequence of steps |
| Agent | Dynamic responses to unexpected inputs and edge cases |

The choice depends on the task, and experimenting with alternative structures helps expose their tradeoffs. LangChain’s initial role was to make that prototyping easier.

[LangSmith](https://www.langchain.com/blog/announcing-langsmith) addresses the accompanying visibility problem. As an application becomes more complex, its final answer tells you less about how it reached that answer. Inspecting the exact sequence of tools and model calls makes it possible to understand and debug the execution itself.

### Inspect the data and the fully assembled prompt

[12:23](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=743s)

Supplying the right context entails familiar data-engineering work: loading, transforming, and transporting data, then observing what moved where. LangChain provides modules for loading and transformation; LangSmith helps reveal what the data looks like when it reaches the model. Two debugging questions remain distinct: did the vector store return the right documents, and did the application transform and format those documents so their contents are clear to the model? Correct retrieval can still be undermined by poor presentation.

Prompt engineering likewise requires looking at the actual input, not just the template. A fully assembled prompt may combine system instructions, few-shot examples, retrieved context, chat history, and previous agent steps. The first call is relatively easy to inspect. After several intermediate steps, the prompt depends on accumulated application state, making it harder to reconstruct by looking at the initial inputs alone.

The playground workflow makes a particular call available for direct experimentation:

1. Locate the model call inside the execution trace.
2. Open that call in the playground with its assembled prompt.
3. Edit the instructions and inspect the resulting response.
4. Swap model providers to investigate whether another model handles that input better.

This lets prompt iteration begin at the point where behavior needs investigation, including deep inside a multistep application.

### A working MVP still needs evaluation data

[14:22](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=862s)

Evaluation encounters two shortages: **data and useful metrics**. Traditional machine-learning development generally begins with a dataset, leaving examples available when evaluation starts. A language-model application can reach a working MVP through zero-shot behavior without collecting an application dataset at all. That makes initial development easier while leaving a gap in the evaluation process.

Building an evaluation dataset therefore becomes deliberate work. Examples can come from manual labeling, selected production traffic, or generation with language models. The dataset is something to build up and iterate on as the application develops, rather than something the MVP necessarily supplied.

Metrics are difficult because large, unstructured responses do not fit many traditional quantitative measures well. Chase sees qualitative inspection—often called a vibe check—as useful for developing intuition about behavior. Exposing model inputs alongside outputs helps explain what happened, even when that inspection does not yield an objective score.

For more systematic evaluation, Chase points to model-assisted judgments and production feedback:

- **LLM-assisted evaluation:** Use a language model to evaluate generated outputs.
- **Direct feedback:** Record explicit user judgments such as thumbs up or thumbs down.
- **Indirect feedback:** Observe behavior such as clicking a suggested link or responding with confusion. These signals may indicate success or failure, but they are suggestive rather than definitive.

Tracking feedback over time and comparing application variants with conventional A/B testing software provides an online view of how the system performs for users.

### The engineering work crosses professional boundaries

[17:01](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1021s)

As these systems grow, their construction spans AI engineers, data engineers, data scientists, and product managers. The right mix of skills for the emerging AI engineer role remains unsettled. Context delivery often needs data-engineering expertise, while the desired reasoning behavior is often expressed through prompts. Product managers and subject matter experts can be especially effective at specifying precisely how an application should behave. Enabling those technical and nontechnical contributors to work together is itself a tooling challenge—one Chase presents as open, not solved.

New model capabilities will continue changing the design space. Chase points to the conference’s **GPT-4V** demonstration as a sign of that change. The closing weather-assistant illustration returns to the original problem with a richer response: it supplies weather information and offers a ride and a coffee order. Those offers depict possible assistance, not evidence that a ride was booked or an order placed.

![Blue slide with a weather question, a hand holding a phone, sparkles, and a response giving San Francisco weather and offering an Uber and a coffee order.](https://ai.engineer/api/talk-essay-media/ad5bd07b2831efa97a1eca41872f817a3088add8472bc007f20541ebbffe224e)

A weather assistant example adds suggestions for a ride and coffee.

Moving from that illustrated experience to a dependable application still requires engineering: connecting capabilities, supplying context, controlling execution, and making the system understandable enough to improve. Supporting that work is the enduring role Chase assigns to the surrounding tools as models continue to change.

## Resources

- [LangSmith's original launch announcement](https://www.langchain.com/blog/announcing-langsmith) — The 2023 introduction to tracing, playground debugging, evaluation datasets and production feedback.
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://arxiv.org/abs/2305.16291v1) — The original paper on a Minecraft agent that accumulates reusable executable skills.
- [Voyager source code](https://github.com/MineDojo/Voyager) — Implementation and setup instructions for Voyager's Minecraft environment and skill library.
- [AutoGPT](https://github.com/significant-gravitas/autogpt) — AutoGPT's project repository, with the original standalone agent retained under classic/ alongside the newer platform.
- [OpenAI's Bet on a Cognitive Architecture](https://www.langchain.com/blog/openais-bet-on-a-cognitive-architecture) — Chase's subsequent explanation of chains, routers, state machines and agent loops, including their control tradeoffs.
- [Evaluate applications with LangSmith](https://docs.langchain.com/langsmith/evaluate-llm-application) — A practical guide to creating datasets, defining evaluators and comparing experiments with the LangSmith SDK.

## Slides

### A chat assistant without real-time weather

Asked for San Francisco’s current weather, the assistant says it cannot access real-time data. The speaker uses this limitation to motivate putting language models inside larger systems.

[0:36](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=36s) [A user asks about current San Francisco weather; the assistant says it cannot access real-time data and suggests checking a weather service.](https://assets.ai.engineer/sites/aiecode2025/sha256/87613304525cc12bd8a730910b28536749e0587732d7d6b924b3ddc024844ab5.webp)

what’s the weather like in San Francisco right now?

I’m sorry but I currently do not have the capability to access real-time data, including current weather information. If you want up-to-date weather information for San Francisco, or any other location, I’d recommend checking a reliable weather website or using a weather app on your mobile device.

If you have any other questions or need assistance with a different topic, please let me know!

Interface text:
H

### A phone among devices and roles

A phone surrounded by devices and roles illustrates the speaker’s vision of personalized assistants built with language models as part of a larger system.

[0:54](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=54s) [A hand holds a phone surrounded by icons of a house, game controller, computer, television, pilot and teacher.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fc85d63a53fa69ccc3899edf3c4f7cf266178546dcb2cab343b3b3cd13fd16e.webp)

### Personalized assistant example

A hypothetical assistant combines the weather with a bike commute and coffee preferences to suggest a rain jacket, an Uber and a hot latte.

[1:03](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=63s) [A weather question and an assistant response suggesting a rain jacket, a ride, and a hot coffee.](https://assets.ai.engineer/sites/aiecode2025/sha256/780de67ebb605f5b6d532c42e696773da9031910310f7207076eccebb7ebb088.webp)

what’s the weather like in San Francisco right now?

It's a rainy day in San Francisco. The temperature holds at 55°F, making it a comfortable, albeit damp. By midday, showers are expected to move in, so I’d recommend wearing a rain jacket.

I know you’ve been enjoying your bike ride to work lately, but would you like me to call you an Uber instead?

I could also call in your coffee order so it’s ready in your building. Should we make it a hot, instead of iced, latte today?

### Instruction Prompting

Instruction prompting gives a model behavioral guidance, like an employee handbook. This example asks a chatbot to be helpful, use a friendly tone, and discuss only LangChain.

[2:09](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=129s) [Instructions to be helpful, use a friendly tone, and discuss only LangChain.](https://assets.ai.engineer/sites/aiecode2025/sha256/bac377493af05c48ddab8a094962f908560ed72fbd345fbe74757f9ac5364d5a.webp)

1. Instruction Prompting

Be a helpful chatbot
Respond in a friendly tone
Only talk about LangChain

### Few Shot Examples

Example exchanges show a LangChain chatbot greeting a user and declining a joke request. The speaker suggests examples can convey tone and structured output when behavior is hard to describe.

[2:54](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=174s) [Two example exchanges show a LangChain chatbot greeting and declining an unrelated joke request.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d3330e4fb6465803f442e8d2982f90dba49aed36f84c66eb80ae5ed99287625.webp)

2. Few Shot Examples

Hi // Hi. I’m a LangChain Chatbot.

Can you tell me a joke? // I only help with LangChain

### Retrieval Augmented Generation

RAG retrieves context for a user question and asks the model to base its answer on it. The example supplies a passage about LangChain as that context.

[3:38](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=218s) [A prompt asks for an answer based on supplied LangChain context.](https://assets.ai.engineer/sites/aiecode2025/sha256/6603c9cae6fd7800f609450d75458632aedb98e84137618c4c14a0916c3c6a91.webp)

3. Retrieval Augmented Generation (RAG)

answer the user question based only on the context below

LangChain is a framework for developing applications powered by large language models. It enables context-aware, reasoning applications. The main value props of LangChain are components and off-the-shelf chains. Components are abstractions for working with large language models, along with a collection of implementations for each abstraction. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not. Off-the-shelf chains are structured assemblies of components for accomplishing higher-level tasks.

### Fine-tuning

Fine-tuning updates a language model’s weights. The speaker sees promise for tone and structured data parsing, using many examples to shape responses.

[4:19](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=259s) [A fine-tuning slide with a stylized neural network diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/d26a5a02acf4b3242702ae7000ba5c1a7436cc7a9b5847e92826dc70cb74bc60.webp)

4. Fine-tuning

### Approaches to LLM reasoning

The table moves from code controlling all three decisions to an agent whose LLM controls them. Routers and state machines let the LLM choose steps; code determines the available sequences, without and with cycles, respectively.

[5:13](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=313s) [A table compares code, LLM calls, chains, routers, state machines and agents by which decisions are made by code or an LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/c9d7a30620af8a93bfb9a4de9fb0d1d37a40d04b3ce97a42bcc92f3392814b43.webp)

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

Legend: code; LLMs

1 Code
2 LLM Call
to single model
one step only
3 Chain
together multiple LLM calls
multiple steps
4 Router
no cycles
5 State Machine
cycles
6 Agent
autonomous

LangChain

### Code

In plain code, the program determines each step’s output, which steps to take, and which sequences are available.

[5:33](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=333s) [A code decision table shows code controlling all three decisions beside a blank terminal window.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6453fc0932ecf5dbb1f895edc570d00c96616bf8929e87d340acb620d155da8.webp)

1 Code

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

LLMs
code

Interface text:
Terminal — login — 80×24

### LLM Call

A single LLM call determines the output of one step. Code determines which steps to take and which sequences are available.

[5:51](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=351s) [An LLM call decision table shows the model deciding the output of one step, with model logos and a blank rectangle.](https://assets.ai.engineer/sites/aiecode2025/sha256/a07497c6a888992541d232c377fb0b3955f5bd180b1c99aae464a3f6e786e1db.webp)

2 LLM Call

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

one step only

LLMs
code

Llama 2
AI
Falcon 180B

### Chain

A chain follows a known sequence: generate a query, look up documents, then generate an answer. The LLM decides step outputs, while code determines the steps and available sequences.

[6:11](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=371s) [A chain runs Generate Query, Lookup Docs, and Generate Answer in sequence.](https://assets.ai.engineer/sites/aiecode2025/sha256/857af5e64eb5920a631aef56b2639318acc73eb86c6d8c2c5515555c0cbfb8d9.webp)

3 Chain

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

multiple steps

LLMs
code

Generate Query → Lookup Docs → Generate Answer

### Router

An LLM chooses which branch to take, such as a prompt, model, or tool. Code defines the available sequences, which have no cycles.

[6:58](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=418s) [An LLM branches to two paths; the upper path continues to an oval. The slide notes there are no cycles.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ec7ead48a5f122814cbadfefea3b48001cbceba5fdc6e1659fdf2986c7cadc7.webp)

4 Router

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

no cycles

LLMs
code

LLM

### State Machine

Code defines the available sequences of steps. The LLM decides each step’s output and which steps to take, continuing the loop or choosing to end it.

[7:55](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=475s) [A state-machine diagram has an entry circle, two unlabeled boxes, and an End box; the slide notes cycles.](https://assets.ai.engineer/sites/aiecode2025/sha256/d90db1bb6c69239a34dfac03a65dd208a53a54aeee257f08fd6a87a1a7c88af8.webp)

5 State Machine

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

🗣️  🗣️  👩‍💻
cycles

🗣️ LLMs
👩‍💻 code

End

[8:02](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=482s) [A state-machine diagram has an entry circle, two unlabeled boxes, and an End box; the slide notes cycles.](https://assets.ai.engineer/sites/aiecode2025/sha256/e048337ca7fe03abc38bce1b9817d83b8c81a0bf49f75708e9d96ea1d753b179.webp)

5 State Machine

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

🗣️  🗣️  👩‍💻
cycles

🗣️ LLMs
👩‍💻 code

End

[8:11](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=491s) [State-machine slide with decision categories and a diagram ending at an End box.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5b82aecf3d88ad0a5c1ba4458c5f1011efef0e7a0899203ad2f5f5ed112bb69.webp)

5 State Machine

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

🗣️  🗣️  👩‍💻
cycles

🗣️ LLMs
👩‍💻 code

End

FIXIE  supabase

### Agent

An agent lets the LLM shape which sequences of steps are available. Actions can feed back to the LLM or lead to an output.

[8:36](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=516s) [Agent diagram showing input, LLM, action, output, and a loop from action back to the LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/76a9f2e6fa50bdc21580fa1c9bef444af41cd876b2d01e286787588ba379f1d7.webp)

6 Agent

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

LLMs
code

Input
LLM
Action
Output

### Agent

The LLM selects steps and actions in a loop, with planning and validation handled implicitly rather than as distinct sequences laid out in code.

[9:03](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=543s) [Agent diagram showing input, LLM, action, output, and a loop from action back to the LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f823b2b84dcf48dc2ddc17444a6df6b93de510dc177e61e6f4cc5c23dfd1e1f.webp)

6 Agent

Decide Output of Step
Decide Which Steps to Take
Determine What Sequences of Steps are Available

LLMs
code

Input
LLM
Action
Output

### Weather Assistant Example

A weather assistant pairs a rainy forecast with offers to call an Uber instead of a bike commute and order a hot rather than iced latte.

[10:21](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=621s) [Example assistant response to a weather question, with suggestions for transportation and coffee.](https://assets.ai.engineer/sites/aiecode2025/sha256/0363fdadf0ab4d4bc78d31462c778117c9d5f22047613915c62e7d2855b97f25.webp)

what’s the weather like in San Francisco right now?

It's a rainy day in San Francisco, with a steady drizzle throughout the day. The temperature holds at 55°F, making it a comfortable, albeit damp, experience. By midday, showers are expected to move in, so I’d recommend grabbing a rain jacket and umbrella.

I know you’ve been enjoying your bike ride to work lately, but would you like me to call you an Uber instead?

I could also call in your coffee order so it’s ready in your building. Should we make it a hot, instead of iced, latte today?

Interface text:
H

### Engineering Behind the Magic

The speaker asks what engineering it takes to build experiences that reason over relevant context.

[10:30](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=630s) [Slide states that a user's magic moment is backed by an extraordinary feat of engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d130f8e0031dbea7977d3fedeabcdc28bf711be1dcf15206ab58f38af3d7d65.webp)

Behind every user’s magic moment is an extraordinary feat of engineering

### Agent Execution Trace

The trace shows two chains around a weather tool call, exposing the steps behind an assistant’s rainy-day response. LangSmith presents this sequence to help developers understand and debug complex applications.

[10:40](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=640s) [Agent execution trace with a weather tool call, model steps, and the resulting assistant response.](https://assets.ai.engineer/sites/aiecode2025/sha256/92dcbc3ca57c4f6c136ea43b9a4486091d46f4d836a35f887672865a82816998.webp)

AgentExecutor

INPUT
input: What's the weather like in San Francisco right now?
YAML

OUTPUT
It’s a rainy day in San Francisco, with a steady drizzle throughout the day. The temperature holds at 55F, making it a comfortable, albeit damp, experience. By midday, showers are expected to move in, so I’d recommend grabbing a rain jacket and umbrella.

I know you’ve been enjoying your bike ride to work lately, but would you like to call you an Uber instead?

I could also call in your coffee order so it’s ready in your building. Should we make it a hot, instead of iced, latte today?

START TIME
10/08/2023, 08:36:47 PM
END TIME
10/08/2023, 08:36:50 PM
TIME TO FIRST TOKEN
N/A
STATUS
Success
TOTAL TOKENS
335 tokens
LATENCY
2.92s
TYPE
Chain

Interface text:
Personal > Projects > default > AgentExecutor
Playground
Add to Dataset
Trace
AgentExecutor · Success
  Chain RunnableSequence
    Chain RunnableMap
    Prompt ChatPromptTemplate
    LLM ChatOpenAI
    Parser OpenAIFunctionsAge...
  Tool get_weather
  Chain RunnableSequence
    Chain RunnableMap
    Prompt ChatPromptTemplate
    LLM ChatOpenAI
    Parser OpenAIFunctionsAge...
Run
Feedback
Metadata

[11:02](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=662s) [A successful agent trace showing two chains and a weather tool call.](https://assets.ai.engineer/sites/aiecode2025/sha256/cdeda597289b7b3efc922e087efde12d5b7303bb0170af1c507e4111136cfeec.webp)

Trace
AgentExecutor · Success
  Chain RunnableSequence
    Chain RunnableMap
    Prompt ChatPromptTemplate
    LLM ChatOpenAI
    Parser OpenAIFunctionsAge...
  Tool get_weather
  Chain RunnableSequence
    Chain RunnableMap
    Prompt ChatPromptTemplate
    LLM ChatOpenAI
    Parser OpenAIFunctionsAge...

Interface text:
Personal > Projects > default > AgentE...

### Inspecting a model call

The selected model call shows the context and weather question sent to ChatOpenAI, followed by its get_weather function call. The trace makes the data reaching the model visible for debugging.

[12:15](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=735s) [A trace view shows system and human messages alongside a weather function call.](https://assets.ai.engineer/sites/aiecode2025/sha256/d686456cb061cdece7576979f92233c686031cee3c3519b81adf2d767d730cf0.webp)

ChatOpenAI

SYSTEM
You are a helpful assistant.

The time is currently: Monday, 8:13am

This is what you know about the human:

Usually bikes to the office, unless it is raining.

Does not have a car.

Likes both hot and cold coffee depending on the weather.

HUMAN
What's the weather like in San Francisco right now?

AI
function_call:
  name: get_weather
  arguments: |-
    {
      "city": "San Francisco"
    }

Interface text:
Personal > Projects > default > AgentExecutor > ... > ChatOpenAI
Playground
Add to Dataset
Run
Feedback
Metadata

Trace
AgentExecutor • Success
  Chain RunnableSequence
    Chain RunnableMap
      Chain <lambda>
      Chain <lambda>
      Chain <lambda>
      Chain <lambda>
    Prompt ChatPromptTemplate
    LLM ChatOpenAI
    Parser OpenAIFunctionsAge...
  Tool get_weather
  Chain RunnableSequence
    Chain RunnableMap
      Chain <lambda>
      Chain <lambda>
      Chain <lambda>
      Chain <lambda>
    Prompt ChatPromptTemplate
    LLM ChatOpenAI
    Parser OpenAIFunctionsAge...

### Editing a model call in the playground

The playground opens a model call for prompt and provider experiments. Here, the weather question produces a get_weather function call for San Francisco.

[13:00](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=780s) [A playground displays model settings, editable messages, and a weather function call result.](https://assets.ai.engineer/sites/aiecode2025/sha256/52502b299936d70e4f5182c2e8a9e8e2369b41146db7c56d72d156eb012317f9.webp)

Editing ChatOpenAI

Settings
Looking for Azure OpenAI?
Specify the deployment and instance name in Secrets & API Keys section.

Provider: ChatOpenAI
Model: gpt-3.5-turbo-16k
Temperature: 0
Maximum Length: -1
Top P: 1
Presence Penalty: 0
Frequency Penalty: 0
Function Call: auto
Functions 1

Input
SYSTEM
You are a helpful assistant.

The time is currently: Monday, 8:13am

This is what you know about the human:

Usually bikes to the office, unless it is raining.

Does not have a car.

Likes both hot and cold coffee depending on the weather.

HUMAN
What's the weather like in San Francisco right now?

Add Message

Latest Run 5 minutes ago 1.04s seq:step:3
AI
function_call:
  name: get_weather
  arguments: |-
    {
      "city": "San Francisco"
    }
YAML

Interface text:
Personal
Back to ChatOpenAI Run
Secrets & API Keys
History
Start

[13:58](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=838s) [A model playground shows settings, an editable weather question, and a weather function call.](https://assets.ai.engineer/sites/aiecode2025/sha256/776b09d97d42c2511ede0d6e5a3aae69d01569d01ca042bb7781dca13f05edce.webp)

Editing ChatOpenAI

Settings
Looking for Azure OpenAI?
Specify the deployment and instance name in Secrets & API Keys section.

Provider: ChatOpenAI
Model: gpt-3.5-turbo-16k
Temperature: 0
Maximum Length: -1
Top P: 1
Presence Penalty: 0
Frequency Penalty: 0
[illegible]n Call
Functions 1

Input
SYSTEM
You are a helpful assistant.

The time is currently: Monday, 8:13am

This is what you know about the human:

Usually bikes to the office, unless it is raining.

Does not have a car.

Likes both hot and cold coffee depending on the weather.

HUMAN
What's the weather like in San Francisco right now?

Add Message

Latest Run 5 minutes ago 1.04s seq:step:3
AI
function_call:
  name: get_weather
  arguments: |-
    {
      "city": "San Francisco"
    }
YAML

Interface text:
Personal
Back to ChatOpenAI Run
Secrets & API Keys
History
Start
AIE
FIXIE
supabase

### Test runs: correctness and latency

The table compares correctness with P50 and P99 latency across test runs. The speaker notes that an LLM application can reach a working MVP without a dataset, making evaluation harder.

[14:25](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=865s) [A test-runs table lists correctness scores, median and tail latency, and recent run times.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3070f974fbcd127e136bc83fdf72392bee20a7b2f1a04ab70c2cc6f64fe6417.webp)

Name | Feedback | P50 Latency | P99 Latency | Most Recent Run
9cf7bd1b53a24deabfe5156597... | CORRECTNESS : 0.76 | 10.17s | 36.95s | 8/10/2023, 1:54:05 PM
29bdfa264c5d4dbe818b3baa3e... | CORRECTNESS : 0.82 | 7.46s | 86.38s | 8/10/2023, 1:50:06 PM
8857a50f727f4e2daa2d0601be... | CORRECTNESS : 0.77 | 3.90s | 314.51s | 8/10/2023, 1:44:03 PM
78815bf71bf34801b72af086047... | CORRECTNESS : 0.33 | 2.52s | 10.78s | 8/10/2023, 1:39:39 PM
6251c6cdc1d340fd8abf000237f... | CORRECTNESS : 0.78 | 4.85s | 58.30s | 8/10/2023, 1:10:02 PM
51abda3d449e454cba3567225... | CORRECTNESS : 0.78 | 4.49s | 17.54s | 8/10/2023, 12:58:17 PM
df21bf41fd774a1d8464b8bf951... | CORRECTNESS : 0.78 | 2.42s | 3.66s | 8/10/2023, 11:49:24 AM
04ceecdc310c4921ab09f51252... | CORRECTNESS : 0.76 | 5.00s | 442.62s | 8/10/2023, 11:28:08 AM

Interface text:
Test Runs
Examples
Search by name...

[16:34](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=994s) [A projected dashboard lists test runs with correctness feedback, latency, and recent run times.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb618b37af5f5d0b2cd6a1a81a1f610c05337b6dc4a1fb7f2d41e426b0a637b4.webp)

Name | Feedback | P50 Latency | P99 Latency | Most Recent Run
[illegible] | CORRECTNESS : 0.76 | 10.17s | 36.95s | 8/10/2023, 1:54:05 PM
[illegible] | CORRECTNESS : 0.82 | 7.46s | 86.38s | 8/10/2023, 1:50:06 PM
[illegible] | CORRECTNESS : 0.77 | 3.90s | 314.51s | 8/10/2023, 1:44:03 PM
[illegible] | CORRECTNESS : 0.33 | 2.52s | 10.78s | 8/10/2023, 1:39:39 PM
[illegible] | CORRECTNESS : 0.78 | 4.85s | 58.30s | 8/10/2023, 1:10:02 PM
[illegible] | CORRECTNESS : 0.78 | 4.49s | 17.54s | 8/10/2023, 12:58:17 PM
[illegible] | CORRECTNESS : 0.78 | 2.42s | 3.66s | 8/10/2023, 11:49:24 AM
[illegible] | CORRECTNESS : 0.76 | 5.00s | 442.62s | 8/10/2023, 11:28:08 AM

Interface text:
Test Runs
Examples
Search by name...

AI Engineer SUMMIT
Presented by AUTO GPT

### Prompt gallery

A prompt gallery organizes shared prompts by use case and model. The speaker raises collaboration between data engineers who bring context and product managers or subject matter experts who specify applications through prompts.

[16:56](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1016s) [A prompt gallery with use-case and model filters, search, and prompt cards.](https://assets.ai.engineer/sites/aiecode2025/sha256/10c96319a67276d789c36ccde57ff301df0c6693c9d5e7e8804138793d581607.webp)

Interface text:
Search prompts, use cases, models...
Recently Updated
Top Favorited
Top Downloaded
Top Viewed

Summarization
hardkothari/prompt-maker
Convert your small and lazy prompt into a detailed and better prom...
ChatPrompt · Updated 3 days ago · 26k · 52k
Try it

Code Understanding
jacob/langchain-tsdoc-comment-generation
Outputs TSDoc comments according to a schema, given some inpu...
Prompt · Updated 3 days ago · 126k · 52k
Try it

Extraction
langchain/entity-memory-extractor
Outputs TSDoc comments according to a schema, given some inpu...
Prompt · Updated 3 days ago · 126k · 52k
Try it

Summarization
hardkothari/tweet-from-text
Generate twitter threads from a text block.
Try it

Use cases
QA over documents 3,224
Interacting with APIs 2,342
Code understanding 2,111
Agent simulations 2,058
Agents 1,279
Chatbots 1,229
Agents 1,112
Code writing 1,010
Autonomous agents 1001
Extraction 989
Multi-modal 923
Extraction 912
Self-checking 909
Tagging 902
Summarization 901
SQL 850

Model
Anthropic: Claude-2 242
Meta: Llama-2-13b-chat 489
OpenAI: gpt-3.5-turbo 1093

Type
ChatPromptTemplate 5,234

### Weather assistant example

The assistant answers a weather question, then uses the rider’s commute and coffee preferences to offer an Uber and a hot latte alongside a rain jacket recommendation.

[18:08](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1088s) [A weather assistant answers a San Francisco weather question and offers transport and coffee suggestions.](https://assets.ai.engineer/sites/aiecode2025/sha256/327f55048590785882b8ca6da535740e0110fada92296d9c74306b5bad767919.webp)

what’s the weather like in San Francisco right now?

It’s a rainy day in San Francisco. The temperature holds at 55°F, making it a comfortable, albeit damp. By midday, showers are expected to move in, so I’d recommend wearing a rain jacket.

I know you’ve been enjoying your bike ride to work lately, but would you like me to call you an Uber instead?

I could also call in your coffee order so it’s ready in your building. Should we make it a hot, instead of iced, latte today?

Interface text:
AI Engineer SUMMIT
Presented by
AutoGPT

[18:13](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1093s) [A weather question beside an assistant response suggesting a rain jacket, a ride, and a hot latte.](https://assets.ai.engineer/sites/aiecode2025/sha256/90cd9a81091a125fbf2193760264e824fe025dbc7fc1ef1d896317ce7941efb5.webp)

what’s the weather like in San Francisco right now?

It’s a rainy day in San Francisco. The temperature holds at 55°F, making it a comfortable, albeit damp. By midday, showers are expected to move in, so I’d recommend wearing a rain jacket.

I know you’ve been enjoying your bike ride to work lately, but would you like me to call you an Uber instead?

I could also call in your coffee order so it’s ready in your building. Should we make it a hot, instead of iced, latte today?

## Full transcript

[0:00](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=0s) [on-hold music] Thank you guys for having me, and then thank you guys for being here.

[0:16](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=16s) This is, this is maybe one of the most famous, uh, screens of, of twenty twenty-three. And, and yet I believe, and I think we all believe, and that's why we're all here, that this is just the beginning of a lot of amazing things that we're all going to create.

[0:33](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=33s) Because as good as ChatGPT is and as good as the language models that underlie them are, by themselves, they're just the start. By themselves, they don't know about current events, they cannot run the code that you write, and they don't remember their previous interactions with you.

[0:50](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=50s) In order to get to a future where we have truly personalized and actually helpful AI assistants, we're gonna need to take these language models and use them as one part of a larger system.

[1:00](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=60s) And that, and that's what I think a lot of us in here are, are trying to do. These systems will be able to produce seemingly, you know, amazing and magical experiences.

[1:11](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=71s) They'll understand the appropriate context, and they'll be able to reason about it and respond appropriately.

[1:18](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=78s) At LangChain, we're trying to help teams close that gap between these magical experiences and, and, and the work that's actually required to get there, and we believe that behind all of these seemingly magical product moments, there is an extraordinary feat of engineering.

[1:34](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=94s) And that's why it's awesome to be here at the AI Engineering Summit. And so I'm gonna talk a little bit about some of the approaches that we see work for developers when they're building these context-aware reasoning applications that are gonna power the future.

[1:49](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=109s) So first, I'm gonna talk about context. A-and when I say context, I mean bringing relevant context to the language model so that it can reason about what to do.

[1:57](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=117s) And bringing that context is really, really important because if you don't provide that context, no matter how good the language model is, it's not gonna be able to, to figure out what to do.

[2:06](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=126s) And, and then, so the first type of context, and probably the most common type of context that we see people bringing to the language model, we see them bringing through this instruction prompting type of approach, where they basically tell the language model how to respond to specific scenarios or specific inputs.

[2:23](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=143s) Um, this, this is pretty straightforward, and I think the way to think about it is if, uh, you have a new employee who shows up on the first day of work, you give them an employee handbook, and it tells them how they should behave in certain scenarios.

[2:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=152s) And, and I equivalate that to kind of like this instruction prompting technique. It's, um, you know, it's pretty straightforward. I think that's why people start with it. And as the models get better and better, this zero-shot type of prompting is gonna be able to carry a lot of the relevant context for how you expect the language model

[2:49](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=169s) to behave. There's some cases where telling the language model is actually quite hard, and it becomes better to give it some few-shot examples. It becomes better to give it examples where you show the language model how to behave rather than just tell it how to behave.

[3:03](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=183s) And so I think a few concrete places where this works is where it's actually a little bit difficult to describe how exactly the language model should respond. So tone, I think, is a good use case for this, and then also structured output is a good use case for this.

[3:17](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=197s) You can give examples of the, of the structured output format. You can give examples of the output tone a little bit more easily than you could describe in language my particular tone.

[3:27](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=207s) The structured output is a little bit-- You can describe structured output, but I think as it starts to get more and more complicated, giving these really specific examples can help.

[3:36](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=216s) The next type of context is maybe the most, um-- You know, it pops to the mind most when you hear of context. I mean, you hear about bringing context to the language model.

[3:44](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=224s) Contrasting this with the first two, retrieval-augmented generation uses context not to decide, uh, how to respond, but to decide kind of like what to base its response in. So the kind of like canonical thing is you have a user question, you do some retrieval strategy, you get back some context, you pass that to the language model, and

[4:01](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=241s) you say, "Answer this question based on the context that's provided to you." Um, and so this is a little bit different from the instructions. It's, it's maybe the same as asking someone to take a test with, uh, like an open-book test.

[4:11](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=251s) You can look at the book. You can look at the answers. And in this case, the answers are the text that you pass in to this context.

[4:18](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=258s) And, and then the fourth way that we see people s- providing context to language models is through fine-tuning, so updating the actual weights of the language model. Um, th-this is still kind of like in its infancy, and I think we're starting to figure out how best to do this and what scenarios this is, this is good to

[4:33](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=273s) do in. Um, s- One of the things that we've seen is that this is good for the same use cases where few-shot examples are kind of good. It takes it to another extreme.

[4:42](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=282s) Um, and so for tone and structured data, uh, parsing, these are two use cases where we've seen it pretty beneficial to start doing some fine-tuning. And, and the idea here is that, yeah, it can be helpful to have three examples of how your model should respond and what the tone there should be.

[4:56](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=296s) But what if you could give it ten thousand examples and it updates its weight accordingly? And so I think for those where, where the output is in a, a, a specific format, and again, you need more examples, you need to show it a lot more than you can tell it, this is where we see fine-tuning starting to

[5:09](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=309s) become helpful. And I think we'll see that grow more and more over time. So we've talked about context. Um, and now I wanna talk a little bit about the reasoning bit.

[5:17](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=317s) And I think this is the most exciting and the most new bit of it as well. And so we've tried to think and categorize some of the approaches that we've seen to allow, uh, these applications to do this reasoning component.

[5:28](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=328s) And, and so we've listed a few of them out here and tried to, tried to discern a few different axes along which they kind of vary. So if we think about kind of like just plain old code, this is kind of like the way things were, you know, in the-- [chuckles] like a year ago, so a long, long

[5:42](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=342s) time ago. Um, and so in code, you kind of like-- You, you-- It's all there. It's declarative; it says what to run. Um, it says what the outputs are, what, what steps to take, things like that.

[5:52](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=352s) We start adding in a language model call, and so this is like the, the simplest form of, of these reasoning applications. And here you're using the language model to determine what the output should be, but that's it.

[6:02](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=362s) You're not, you're not using it to take actions yet, nothing fancy. You're just using it to determine what the output should be, and it's just a single language model call.

[6:07](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=367s) So you're providing the context and, and then you're, uh, returning the output to the user.

[6:13](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=373s) If we take it up a little bit, then we start to get inter-- like chain of, of language model calls or a chain of language model call to API back to language model.

[6:22](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=382s) And so this can be u- this is again used to, uh, decide the, the, the steps of the output. Um, a-and here there's, uh, multiple calls that are happening, and this can be used to break down, um, more complex task into individual components.

[6:37](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=397s) It can be used to insert knowledge dynamically in the middle of kind of like one language model call. Then you go fetch some knowledge based on that language model call, and then you do another one.

[6:45](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=405s) Um, but importantly here, the steps are known. You do this, and then you do this, and then you do this. And so it's a chain of events. And that starts to change a little bit when you use a router.

[6:54](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=414s) Um, and, and so in here, you're now using the language model call to start determining which steps to take. So that's the big difference here. It's no longer just determining the output of the system, but it's determining which steps to take.

[7:04](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=424s) And so you can use it to determine, uh, which prompts to use, so route between a prompt that's really good at math problems versus a prompt that's really good at, at r- uh, writing English essays.

[7:14](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=434s) You can use it to route between, uh, language models. So one model might be better than another. You might wanna use Claude because of its long context window, or you might wanna use GPT-4 because it's really good at reasoning.

[7:24](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=444s) And so having the language model look at the question and decide whether it needs to reason or whether it wants to respond in a long-form fashion, you can determine which branches to go down.

[7:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=452s) Or I think m-- uh, another common use case is using it to determine which of several tools to take. So do I wanna call this tool, or do I wanna call this tool?

[7:39](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=459s) And what sh-- what should the input to that tools be? And so we have this router here. And I think, um, before going on to the next step, the, the main thing here that distinguishes it from that step is that there's no kind of like cycles.

[7:50](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=470s) You don't kind of get these, these loops. Um, uh, you're just choosing kind of like which branch to go down. Once you start adding in these loops, this is where we see, uh, a lot more complex applications.

[8:01](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=481s) Um, this is, this is-- these are things that we often see being called agents kind of like out, out in the wild, and it's essentially kind of like a while loop.

[8:09](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=489s) Um, and then in that loop, you're doing a series of steps, um, and the language model is determining which steps to do. And then at some point, there's a point where it can choose whether to end the loop or not.

[8:18](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=498s) Um, and if it ends the loop, then you finish and return to the user. Otherwise, you go back and, and, and continue the loop. And so here you get the language model deciding what the outputs are.

[8:26](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=506s) It decides what steps to take, and you do have these, these cycles. Um, the last thing, um, and I think the, the-- t-this is, this is largely what we would describe as kind of like what Auto-GPT did that took the world by storm, is this idea of an agent, um, where you kind of like remove a lot

[8:44](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=524s) of the, um, the, the kind of like guardrails around what steps to take. So here, the sequences of steps that are available are, are almost like determined by the LLM.

[8:55](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=535s) And what I mean by this is that here's where you can start doing things like adding in tools that the, that the language model can take. So if you guys are familiar with the Voyager paper, it, it starts adding in tools and building up a skill set of tools over time.

[9:07](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=547s) And so some of the actions that the language model can take are, are dynamically created. Um, and then I think the other big thing here is that you remove some of the scaffolding from the state machines.

[9:17](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=557s) Um, so s-some of the, uh-- If I go back a little bit. So a lot of the-- these kind of like cycles that we see in the wild break things down into discrete states.

[9:26](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=566s) The most common one that we see are kind of like plan, execute, and validate. So you ask the language model to plan what to do, it then goes do it, and then you validate it often with a language model call or something like that.

[9:36](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=576s) And I think the big difference between that and then the autonomous agent style thing is that here you're implicitly asking the agent to do all of those things in one go.

[9:45](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=585s) It should know when it should plan, it should know when it should, uh, validate, and it should know when it should kind of like determine what action to take.

[9:51](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=591s) And you're asking it all to do that implicitly. You don't have these kind of like distinct sequences of steps laid out in the code.

[10:00](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=600s) And so this is a little bit about how we're thinking about it. I think the, the thing to, uh, th-the thing that I like to say when saying this as well, which goes back to the beginning, is that the main thing that we think is it's still just extremely early on in the space.

[10:13](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=613s) We still think it's the beginning. And this could, you know, in, in three months be kind of irrelevant as the space progresses. So I would just keep that in mind.

[10:21](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=621s) If we think about kind of like some of the magical experiences like this where it can reason over the relevant context, what is it gonna take to kind of like build it under the hood?

[10:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=632s) What is the engineering that's gonna go into all these seemingly magical experiences? And so this is an example of, of what could be going under the hood of something like this.

[10:43](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=643s) It's gonna be a challenging experience to build these complex systems, and that's why we're building some of the tooling like, like this, what you see here, to help debug, understand, and iterate on these systems of the future.

[10:56](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=656s) And so what exactly are the challenges associated with building these complex context-aware reasoning applications? The first is kind of just the orchestration layer. So, uh, figuring out which of the different reasoning, kind of like cognitive architectures you should be using.

[11:13](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=673s) Should you be using, uh, a simple chain? Should you be using a router, a more complex agent? And I think the thing to remember here is that it's not necessarily that one is better than the other or superior to the other.

[11:23](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=683s) They all have kind of like their pros and cons and strengths and weaknesses. So chains are really good because you have more control over the sequence of steps that are taken.

[11:31](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=691s) Agents are better 'cause they can more dynamically react to unexpected inputs and handle edge cases. And so being able to choose the, the right cognitive architecture that you want and being able to quickly experiment with a bunch of other ones are part of what inspired the initial release of LangChain and, and, and kind of how we aim

[11:48](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=708s) to help people, uh, prototype these, these types of applications. And then LangSmith, which is this thing here, provides a lot of visibility into what is actually is going on as these, uh, as these applications start to get more and more complex, understanding what exact sequences of, of tools are being used, what, what exact sequences of language model

[12:07](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=727s) calls are being made becomes increasingly important. Another big thing that we see people struggling with and spending a lot of time on is good old-fashioned data engineering. So a lot of this comes down to providing the right context to language models, and the right context is often data.

[12:23](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=743s) And so you need to have ways to load that data, you need to have ways to transform that data, tr- transport that data, and then you often wanna have observability into what exact data is getting passed around and where.

[12:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=752s) And so LangChain itself has a lot of open source kind of, like, modules for loading that data and transforming that data. And then LangSmith we often see being really useful for debugging what exactly does that data look like by the time it's getting to the language model.

[12:46](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=766s) Have you extracted the right documents from your vector store? Have you transformed them and formatted in the right way where it's clear to the language model what's actually in them?

[12:54](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=774s) These are all things that you're gonna wanna be able to, to debug so there's no little small errors or small issues that pop up.

[13:02](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=782s) And then the, the third thing that we see a lot of people spending time on when building these applications is just good old-fashioned prompt engineering. Um, so the main new thing here is language models, and the main way of interacting with language models is through prompts.

[13:14](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=794s) And so being able to understand what exactly does the fully formatted prompt look like by the time it's going into the language model is, is really important. What d- like, how are you combining, uh, the system instructions with maybe the few-shot examples, um, any retrieved context, the chat history that you've got going on, any previous steps that

[13:33](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=813s) the agent took? What does this all look by the time it gets to the language model? And w- and, and what does this look like in the middle of this complex application?

[13:40](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=820s) So it's easy enough to, to kind of, like, test and debug this if it's the first call, the first part of the system, but after it's already done three of these steps, if you wanna kind of, like, debug what that prompt looks like, what that fully formatted prompt looks like, being able to do that, um, it becomes

[13:54](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=834s) increasingly difficult as the systems kind of, like, scale up in, in their entangledness. And so we've tried to make it really easy to, to hop into any kind of, like, particular language model call at any point in time, open it up in a playground like this so you can edit it directly a- a- and experiment with that

[14:10](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=850s) prompt engineering and, and, and go kind of like, uh, change some of the instructions and see how it responds or swap out model providers so that you can see if another model provider does better.

[14:22](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=862s) Another big challenge with these language model applications, and it's probably worth a, a talk on its own, is evaluation of them. And so I think evaluation is really hard for a few reasons.

[14:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=872s) I think the two primary ones are a lack of data and a lack of good metrics. So comparing to traditional kind of, like, data science and machine learning, with those you generally started with a data set.

[14:42](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=882s) You needed that to build your model, and so then when it came time to evaluate it, you at least had those data points that you could look at it and evaluate on.

[14:49](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=889s) And I think that's a little bit different with a lot of these, um, LLM applications because these, these models are fantastic zero-shot kind of, like, l- learners. That's the-- that's kind of, like, the, the whole exciting bit of them.

[15:00](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=900s) And so you can get to a working MVP without building up kind of, like, any data set at all. And that's awesome, but that does make it a little bit of a challenge when it comes to evaluating them because you don't have these data points.

[15:11](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=911s) And so one of the things that we often encourage a lot of people to do, and try to help them do as well, is build up these data, these data sets and iterate on those.

[15:20](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=920s) And those can come from either labeling data points by hand or looking at production traffic and pulling things in, um, or auto-generating things with, with LLMs. The second, um, big challenge in evaluation is lack of metrics.

[15:33](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=933s) Um, I think most traditional kind of, like, quantitative metrics don't perform super well for, for large unstructured outputs. A lot of what we see people doing is still doing a kind of, like, vibe check to kind of, like, see how the model's performing.

[15:47](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=947s) Um, and, and as unsatisfying as that, as that is, I still think that's probably the best way to gain, uh, kind of like, um, intuition as to what's going on.

[15:55](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=955s) And so a lot of what we try to do is make it really easy to observe the outputs and the inputs of the language models so that you can build up that intuition.

[16:03](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=963s) Um, in terms of more quantitative and systematic metrics, we're, we're very bullish on LLM-assisted evaluation, so using LLMs to evaluate the outputs. Um, and th- and then I think maybe the, the biggest thing that we see people doing in production is just keeping track of feedback, um, whether it be direct or indirect feedback.

[16:21](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=981s) So do they leave kind of, like, a thumbs up or a thumbs down on your application? That's an example of direct feedback where you're gathering that. An example of indirect feedback might be if they click on a link or that that might be a good thing that you provided a good suggestion.

[16:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=992s) Or if they, if they respond really confused to your chatbot, that might be a good indication that your, uh, chatbot actually did not perform well. And so tracking these over time and, and doing A/B testing with that using kind of, like, uh, traditional A/B testing software can be, can be pretty impactful for gathering a sense online of

[16:49](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1009s) how, of how your model's doing. And then the last interesting thing that we're spending a lot of time thinking about is collaboration. So as these systems get bigger and bigger, they're doubtless gonna be a collaboration among a lot of people.

[17:01](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1021s) Um, and so who exactly is, is working on these systems? Um, is it, is it all AI engineers as we're here today? Is it a combination of AI engineers and data engineers and data scientists and, and product managers?

[17:13](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1033s) And I think one of the interesting trends that we're seeing is it's still a little bit unclear what the best skill sets for this new AI engineer-type role is.

[17:22](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1042s) And, and there could very well be a, a bunch of different skill sets that are valuable. So going back to kind of, like, the two things that we see making up a lot of these applications, the context awareness and the reasoning bit.

[17:32](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1052s) The context awareness is bringing the right context to these applications. You often need kind of, like, a data engineering team to get in there and assist with that. The reasoning bit is often done through prompting, and oftentimes that's best done by non-technical people who can really outline kind of, like, the exact specification of the app that they're

[17:48](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1068s) building, whether they be product managers or subject matter experts. And so how do you enable collaboration between these two different types of folks, and, and what exactly does that look like?

[17:57](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1077s) I don't think that's something that anyone kind of, like, knows or definitely has solved, but I think that's a really interesting trend that, that we're thinking a lot about going forward.

[18:07](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1087s) And so I think, like, the, the, the main thing that I wanna leave you all with is that the big thing that we believe is that it's still really, really early on in this journey.

[18:18](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1098s) It's just the beginning. As crazy as things have been over the past year, they're hopefully gonna get even crazier. You saw an incredible demo of, of GPT-4V. Things like that are gonna change it.

[18:28](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1108s) And so we think behind all of these things, it's gonna take a lot of engineering, and we're trying to build some of the tooling to help enable that. And I think you guys are all on the right track towards becoming those types of engineers by being at a conference like this.

[18:41](https://www.youtube.com/watch?v=cwjs1WAG9CM&t=1121s) So thank you, Swyx, for having me. Thank you guys for being here. Have a good rest of your day. [audience applauding] [upbeat music]
