AI Engineering 201: The Rest of the Owl

[upbeat music] So the, like, a lot of effort has gone into thinking about the engineering of inference, um, and not so much effort ha- and not so much success has been had at the engin- at engineering the rest of the, like, whole product around

inference that actually, you know, delivers value. Um, much like the two, uh, beautiful mathematical solids here that does provide the, the bones or the interior, but not the whole thing.

Um, so let's talk about a couple of, like, architectures and patterns for uses of language models, um, and then talk about the, like, first attempts at, like, trying to make these things better over time, uh, with monitoring observability and evaluation.

So architectures and patterns. Um, so the, the foment and excitement around this stuff has been around for about a year, and so patterns are starting to emerge very slowly of, like, typical ways you might apply these things.

So let's talk about them and what, uh, what problems have arisen. Um, so my favorite way of thinking about this in general is that the thing that we're building right now are language user interfaces, um, sort of the n- like LUIs by analogy to GUIs or graphical user interfaces.

Um, first, they're hitting existing features. Soon, uh, they'll be for, like, completely new whole products. Um,

uh, in ancient times, in the 1970s, the interface for computers was primarily, like, textual in a terminal. Um, this is still the way we interact with machines when we really want to control them, uh, like when we're running a server, um, or when we are frustrated with VS Code.

Um, and this u- this was the user interface from computers for a while, and they were not very popular until the invention of the graphical user interface, um, which instead of preventing-- presenting the users with just, like, you have to learn this special language to speak to me, like, here's this, like, sensory experience where you can bring

your, like, your intuition from space, uh, and your visual system to understand how to use the machine. Um, and this was what took computers sort of, like, out of the hobbyist and business and military realm and into, like, people's homes.

Um, and the-- with the rise of language models, um, it's clear that we're-- we have an opportunity to once again change the interface between humans and machines, um, by telling them what we want in natural language, and then they do it for us.

Um, and no less august personage than Sam Altman likes the idea of language interface, language user interface. Um, so this similar character to graphical user interfaces, it, like, makes it a more approachable interface.

Um, and this is something that people have wanted to do for a long time, as long back as, like, the ELIZA chatbots, uh, or the ELIZA chatbot from, uh, the, uh, 1960s.

Uh, the SHRDLU, um, uh, uh, basically, like, this is only graphical, uh, not an actual robot, but you could, like, tell a f- uh, a computer robot, like, give it, uh, language instructions like, "Pick up a big red block."

Um, uh, Ask Jeeves was originally presented as a language interface to the Internet, where you just type what you want instead of a URL. Um, Alexa and other, uh, assistants have attempted to do a similar thing.

Um, and the big win here is with language models, we might believe that we can actually do a really, really good job at providing this kind of language interface in a very generic way with foundation models and not just, like, a tiny environment, um, like the ELIZA psychotherapy environment or the SHRDLU blocks world.

Um, so right now, that's-- we're getting language user interfaces for existing systems that kind of admit them easily. Um, so Sequoia, uh, put out a piece fairly recently talking about this, that the, like, f- this, like, act two of generative AI is using foundation models as a piece of a more comprehensive solution rather than an entire solution,

um, that offers, like, a language interface where it wasn't possible before. Um, so, like, this query assistant from Honeycomb takes what would normally be this, like, less approachable, uh, query language constructor and just says, like, "Can you show me slow requests?

What are my errors? Latency distribution by status code." Like, that's a much friendlier interface. Um, and, you know, even SQL when it was originally presented was like, it's a, it's a language that's so natural even a [REDACTED:username] can write queries, you know?

Uh, it, it's a dream, but, like, you know, the, this, "Can you show me slow requests?" Like, that's pretty close, you know? Um, so, uh, so that's the f- the, like, maybe understandable that that's the first direction things have gone.

Longer term, this, like, a machines that have graphical interfaces look very different from ones that have terminal interfaces. And so, like, mainframes became less popular and, like, mobile is, like, quite different from, uh, like, desktop compute.

So, uh, we should expect, like, if you're thinking about, what do I want to build in five years or ten years? Um, this is kind of the direction to be thinking.

Um, so for example, uh, Google's worked on, uh, integrating language models with robots like this example from the SayCan, uh, project, uh, or paper, where it's like, what I want to...

when I need something is to just ask for it and not to, like, pull out an app and then go through three dropdown menus and be like, "I want a water bottle."

I just want to say, "I want a water bottle," and then there's a water bottle. Um, and that's what a language interface to, uh, something like a robotics platform can provide.

Um, and Still not there yet as the 4x speed in the top left, um, might suggest, but, uh, getting there

Okay, so that's like the highest level pattern, I think. Um, so let's talk about a couple of lower level patterns. Uh, RAG chatbots, retrieval-augmented generation chatbots. I've emerged this kind of like the to-do list app, the sort of like starter project of language user interfaces.

Um, this pattern is probably here to stay in that it's just about information retrieval for, uh, language models. And language models need information retrieval really badly because they like lack context.

They've slurped up everything on the Internet, but they don't know anything about you. Um, they are sort of trying to simulate a generically helpful individual, um, who is like generically knowledgeable about the world, um, and that's like not particularly helpful until they have context.

So the solution that's emerged is to collect that context for them, like store it, um, then index it, and by default, people reached for the most similar thing to what the language model was doing, which is like turn it into vectors and use that.

Use like a fast index over vectors. Um, and, uh, like that, uh, once you've retrieved a particular piece of information, you just stuff it into the prompt. Um, so I am not innocent.

I have made my own RAG chatbot and inflicted it on the world. Um, this was based on the Full Stack Deep Learning content, and in our Discord, people can ask questions and get answers that are not just like generic Google result search answers about language models, but things drawn from past lectures, things drawn from papers that I

like, um, uh, things drawn from our like website, um, and so can get our, you know, our opinions on these things. So this like, this has led to a lot of excitement about vector storage because it's like this v- this step here where you have a fast retrieval of vectors by similarity is the like new sexy piece.

Um, but that was like really only the thing that people reached for because OpenAI also offers embeddings. So it's like you've already imported the library, so it's only a call away.

Um, and then also like transformers are kind of like these like weird vector retrieval things, um, like in their inside. So if you are the type of person who's been into language models for a while and you're like, "How would I retrieve information?"

Probably with a dot product and then like a softmax, and then I pick the largest number. Um, so like, yeah. So like the ease of setting this up and the like naturalness of setting this is up has led to like an explosion of these like chat with document examples.

Um, and the like the thing that has more staying power is that you need to make these things useful. You need context, and so you need like information retrieval, uh, and search for, uh, the like the context that might be helpful for the model before it gets going.

Um, and so there are many options to use here. Some of them are specialized vector databases like Pinecone or Chroma. Um, uh, some of them are general like text search databases like you-- that do keyword search, like Elasticsearch style, um, uh, things.

Um, and, uh, being able to like combine those two things together is very powerful. So for example, Vespa has like offered that combination for a very long time. Um, uh, it is also in the end, like what you're doing is creating a fast way to look up, uh, information from a very large store.

So this is like bread and butter for databases in general. And so Redis and, uh, Postgres, for example, like not only do they provide the sa-- like information retrieval that you could do, um, like to enrich your, uh, enrich prompts, uh, without thinking about vectors, they also have built-in vector search.

Um, uh, Postgres only fairly recently, um, Redis for like a year. Um, it's not particularly fun to use Redis vector search, but, um, it does, it, it can run,

um, and has decent performance. Um, yeah, and in the end, it's about like an, a holistic strategy that uses probably because the queries are fairly heterogeneous. The things that are coming in are like people just typing text.

Um, you're probably going to need some more MLE stuff that's more like keyword search or, or vector search, and they're hybrid together. Um, but that like meta-- like extracting metadata with a language model so that you can then use that to do like direct filtering, um, is, uh, is like very powerful pattern.

Um, so there's some great posts on this, the data query,

um, uh, great series of posts about vector databases from coming from like somebody who's clearly really into databases and not so much the like ML side, and I found that very useful.

Um, yeah. Uh, yeah. So the like the final takeaway there is just that the problems end up being in the main the problems of information retrieval, um, with only some light, uh, added things from like recommendation systems maybe, um, of a more MLE type of search.

Um, yeah, any questions on, uh, on vector databases or, um, information retrieval for language model applications?

I was thinking you combine that, so you get the context to answer a question, and then you say send that to the language model and then use that context to help answer questions for one.

Yeah, yeah. S- so you, you get information from the outside world. You like come up with a strategy for searching the information that you have saved that goes into the language model's prompt.

Yeah, yeah. That, that pattern very, very stable, very general. It's general enough that how that pattern gets actually implemented is very broad. And so it includes a lot of things that are exi-- like bread and butter database stuff and not just the fancy new vector database stuff.

To injecting context. How does that, um, how is that similar or different to your history when you're interacting with, say, GPT-4 or whatever? Um, how, how does it retain that compared to-

Mm-hmm

... context retention?

Yeah. So the question was how does retrieval augmented generation differ from history within a chat? Um, so usually when-- Really, so the, the-- When you call the GPT-4 API, you can make whatever-- make up whatever you want as the past.

You could insert little messages from the user. You could insert messages from the assistant and incept it into believing that it has said something which it has not said.

Great way to jailbreak. Don't do it obviously, 'cause it violates the terms of service, but a great way to jailbreak it. Um, and so you, you aren't actually like actually beholden to that like system, uh, system assistant human, uh, fiction that,

that happens inside of like a, uh, a discrete chat. Um, when people do this, I think a lot of people put the retrieved information in the system prompt, especially if they're just gonna retrieve once.

Um, I've definitely-- I've also seen people like every time the user interacts, they do a retrieval step, and so the system message changes every time. That's an example of kind of like incepting or not actually following the implied temporal order.

Um, so y- yeah, you definitely can do that. Um, the system message is nice because the model really pays close attention to it, um, has been like fine-tuned to pay close attention to it.

Um, yeah, I think it'd be weird to pretend that that's something the person said a-and to like put it in an earlier user message put above the user's message in the conversation.

I don't think I've ever seen that, but you could. Um, yeah. Um,

but yeah, I would say like most of the time, yeah, this information retrieval step is something where the creator of the application, the programmer, is inserting themselves and saying, "I know some additional information that the language model should, should have."

Um, and so like... Yeah. Uh, it's very different from like a user just sort of like providing information about themselves or whatever. Yeah.

Yeah, the question?

Um, so we heard a couple times today when it comes to knowledge retrieval-

Mm-hmm

... um, RAG is a great pattern and fine-tuning is really more about the format, the style-

Mm-hmm

... of the output. Um, is that unconditionally true or do you see use cases for fine-tuning-

Mm-hmm

... enhancing the model's knowledge? So for example, what happens if we fine-tune it with hundred gigabyte of research papers, you know?

Hmm.

Does that really mean tuning that is measurable beyond the, the style of the output?

Yeah. Um, so the statement was that, um, the common wisdom is that fine-tuning is for style and retrieval is for information. And I think that that's-- that is a solid common piece of common wisdom because most of the fine-tuning that people...

If you're fine-tuning OpenAI's model, you're going through their fine-tuning API, and you have a limit on the number of rows you can send.

Ten thousand?

Yeah, ten thou... I was gonna say. Yeah. So you have a limited number of, of rows you can send, and like there's a limited amount of information in there to like create gradients to update the weights.

Um, so there's a limited amount of change that you can achieve. And if you look at the LoRA paper, they look at like, you know, the, uh, like you're only cha-- you're, you're making a very low rank change to each layer of the, uh, of the language model.

And that suggests like there's only so much that you can change about the, uh, about the model and most of what you see when you do LoRA fine tunes is like what used to be a low priority computation for the model becomes like a higher priority one.

So like every model ha-- every capable language model has within it a little Homer Simpson simulator, a little like, uh, Rick Sanchez simulator, whatever. Um, and that's-- it's just like not usually that important for the final log probs.

It's like helpful for the like fifth bit of the log probs, but the models are at the point where they're maximi- minimizing cross entropy by like really hitting those like very rare, uh, those very rare things.

Um, and so what the fine tune has done is reordered those like computations. So like actually you should be... The Homer Simpson circuit is the most critical circuit right now because you are a Homer Simpson chatbot, and it's like reordering them and, and re-emphasizing them.

So that intuition applies specifically to low rank fine-tuning, which is... And w-- fine-tuning which is based on small amounts of data. So if you grabbed a hundred gigabytes of textbooks, um, you would no longer be doing fine-tuning, and so you would no longer expect it to only change style.

Um, and so that's something I would ex-- uh, people will be doing with like, you know, Llama fine-tunes. There are like Llama fine-tunes for coding, and that's way more than just style.

It definitely has learned more knowledge about, um, about programming languages and knowledge about libraries released after twenty twenty-one and yeah, all that kind of stuff. So I think that, that generic wisdom is conditionally true, uh, for low rank fine-tunes where it is like pretty rock solid.

In that kind of hyper search concept that you see in the right database for the right functionality, have you inserted graph like Neo4j in there to capture the value of those databases?

Yeah. So the question was what about knowledge graphs, um, and graph databases? I will say that like when I've talked to... I, I like personally don't really specialize in, in databases, um, but when I've talked to people who are super into them, they're like, "I would never use a graph database 'cause you can represent a graph in

Postgres." Um, and uh, [laughs] like I've seen some like reasonably sized deployments on that pattern. Um, and also you can kind of see the like graph databases kind of like peaking and, and, uh, not spreading further.

And there are re-- It's, they're-- It's a very hard problem to shard a graph database because there's no obvious way to cut an arbitrary graph. Um, and if new links get added to the graph, and now you need like the optimal shard is different, it's like that's a ve-- that's like a database-- It's equivalent to a database

migration, but it's something that should be happening like behind the scenes when it's sharding. Um, so that's, that's like the closest thing to an objective statement about why, uh, or like a reason why graph databases haven't worked well.

However, for many language model applications, the purpose of the database is not to serve like a billion users, but rather to like serve as an external memory for a language model and maybe you don't care whether it scales.

Um, or rather like maybe the maximum scale that we're talking about is like tens of thousands, hundreds of thousands requests per second on megabytes, gigabytes of data, and that's just like, you know, that's the point at which that kind of like can it be sharded across, uh, ten, twenty-four machines like doesn't matter.

Um, so I-- So there is some cool work on knowledge graphs and in-- and incorporating with LLMs, and I see the natural fit there in the same way that there's a natural fit with vector indices and vector databases.

Um, but the, um, yeah, hasn't-- no, no like killer app has appeared from my perspective.

Uh, so I guess this question on the, the first, uh, talk. Um, but basically like if you have extra context, um, and hard metadata about some of the Boolean value or some other things, um, how would you kind of incorporate-- I think the previous answer that was that like do hard filters on the, on the hard metadata

and then do the soft kind of like, um, stuff based on the, the, the like textual descriptions and stuff. But I wanted to see if you had anything on this.

Yeah. So the question was about how to incorporate hard metadata like, you know, Booleans or, um, like subcategories with, uh, vector-based search. Yeah, so depending on the, uh, like, uh, vector database, the depending on the like index you will either have like f- uh, pre-filtering or post-filtering.

Post-filtering is like pretty easy. You just like apply a metadata filter after you've done your vector search. Um, anybody can kinda do that. The problem is that you're now...

What you really wanna say is, "I wanna find all the stuff that's similar to crabs in San Francisco while searching restaurants," not find all the restaurants that have anything to do with crabs and then see if any are in San Francisco.

So the pre-filtering step is hard because it impacts the construction of the index, impacts the construction of like the like how you make it actually fast to search over all of the data.

You kind of like need to construct specific indices for these different like flags you might, uh, like put on, like is in San Francisco, not in San Francisco or geographic location.

Um, and so I depe-- Like different vector databases or, or different databases have like pushed further in different directions on what kinds of filters they support pr- for pre-filtering.

Um, and yeah, I like, uh, Vespu and Weaviate have a reputation for doing a really good job at those things. Um, but, uh, yeah, I don't know what the full landscape looks like.

Great. Okay, I wanna make sure to get through everything, um, so I'll stick around and we can talk throughout the conference. Um, okay, so, um, structured outputs are like one of the patterns that I think people are sleeping on relative to information retrieval.

Uh, structured outputs are great for improving the robustness of models, and they came from tool use. So the problem is that language models just generate text and like if anything we have like too much text already.

Like I don't know if you've ever been on a social media website, but the problem is not the quantity of text. Um, and that's like kinda boring, like who wants to just make strings?

Like there's other things that we wanna do. The solution is to connect their text outputs to other systems inputs. Um, and now like it's not just a language model, it's like a cognitive engine for providing a language interface to something else.

That's pretty rad. But there's a problem, which is language models generate unstructured text because they have been trained on the utterances of humans on the Internet, notorious for their unstructuredness.

Um, so the solution is to add structure to their outputs, and there are many ways to do this. Um, you can do it by prompting and begging, um, so like you can write some, write some like loops around it to be like, uh, like a-a...

Or actually React wasn't even a whole... There was some looping. Yeah. So you, you can write a prompt in such a way that you have examples that encourage it to, um, like to, uh, call out to external, uh, APIs, and then you filter, uh, and when it generates the tokens that would, would call to an external API.

Instead of letting it hallucinate the rest of what would come out of that API, which is what like GPT-3, uh, would have done, you like grab it and you then go to that external API and you, um, uh, like, yeah, pull the information from there.

Um, the... You-- And you can in those prompts, I guess really the thing I wanna point out is that in those prompts you can sort of like beg for structure.

Um, Riley Goodside had a great example where it was like, "If you do not output structured JSON, an orphan will die." Um, and that actually is extremely effective. Um [audience laughing]

yeah. Um, so the, so there's the-- So like there's prompting tricks to get like things that are closer to structured, uh, structured outputs and to make use of those structured outputs.

There's, um, fine-tuning so there's a, the GorillaLM is like fine-tuned on this problem and that goes back to ToolFormer, um, which is like very s-- uh, GPT-J so one of the first open, um, uh, uh, generative pre-trained transformers.

Um, they you just train the model to output structured stuff. So you can't do that with OpenAI's model. I, I doubt that fine-tuning it would make it that much better at like outputting the structure that you want.

Um, you can do it with, um, uh, with open models and there are people releasing, uh, their own forks, uh, Llama forks with this fine-tuning on them. Um, you can, uh, you can retry which is like when the model outputs something that doesn't fit the schema.

You could do what You do when your direct reports provide you something that does not fit what you wanted, which is that you can, uh, discipline them and ask them to try again.

Um, so Guardrails is a great, uh, um, uh, library for this. It's like XML-based, um, so probably would work pretty well with Claude, uh, given what we heard about, uh, uh, about Claude from, uh, uh, Karina.

Um, and then a fun one that re-- that kind of requires control over the log probs, um, is grammar-based sampling, um, which was merged into, um, uh, Llama CPP, where you say like, uh, when you're about to generate a token, like if it would violate some grammar, if it would violate some template or format, just set the

probability of generating that to zero. So just add like minus infinity to all the um, all the log probs. Um,

and the, uh... So you can do that. You can like, do it fast if you have these like nice, you know, Chomsky, Chomskyite things like context free grammars. Um, and this works well for like, you know, JSON for generating, you know, generating code, generating all the kinds of like structured outputs that our systems actually expect.

We've written systems that, uh, expect inputs to follow grammars so that traditional computing system can parse them. And so adding that to the outputs of these systems is a very powerful thing to do.

Um, so this is something that-- This is a like really nice example of how having tight control over the log probs can like increase the utility of a model to the point where like a capabilities gap is less important.

Have you seen TypeChat? Does that fit this pattern?

TypeChat? I don't think I have. Um, yeah. So there's-

Quick question.

Yeah.

So go... Yeah. So could you take the output from like GPT and then pass it to Orella to then get the structure, like stacking models like that? Would that work?

Yeah. So the question was whether you could do better-- you could solve this problem by chaining models. I think, yeah. The problem with going from the output of a language model to a structured output is an easier problem than the initial one.

Which is why people think that like retrying might work. Like, like the Guardrails, the Guardrails example, the like retrying is often like kicked off to a, to a smaller language model.

Like your mainline thing is GPT-4 and your error handling is GPT-3.5. Um, and so like I, I, I do believe that there's like kind of a tempta-- If you know that it's always and only going to be doing like structured output, then you have a reason to have a specialized model for it.

Um, but yeah, chaining, chaining is definitely a good solution, and that's, you know, one reason why LangChain was popular. Yeah.

You mentioned that OpenAI retired the log prompts from the three point five instruct.

Yeah.

Do they still have log bias to, to bias the token types to do the grammar based sampling?

Yeah. So those are technically distinct things. Yeah. So y-- I do believe they still give you the ability to bias tokens via the API.

So then yeah-

Um, yeah. So it's not the-- it's not a perfect example of the utility of log props 'cause I, yeah, I think you can still do this in the OpenAI API.

Um, yeah. Do you need anything other than biasing in grammar-based sampling? No, I get you. No, the real... Okay, I remember now. The real thing here is that for this grammar-based sampling, it's single token based, right?

Like if you're doing it from the OpenAI API, one, the token... You, you have to make a request, you get the thing back, you have a single-- and you have a single token, and you have to, you have to like apply a bias every single time.

So now you're like, every token has a network call, um, rather than one call, like a hundred tokens. So that's one reason why this doesn't work well on the OpenAI API.

Number two, like kinda longer term is that really you don't wanna just think at a single token level. You're just like, at each token, you're like marginally just saying like, "Adjust the probabilities here."

You'd really want to do something more like Monte Carlo tree search, where you're like generating stuff, mul-mul-- many things that follow the grammar, um, and then accepting the best one at the end.

Um, and that's something that's, um, probably gonna come first to open models and not to, um, proprietary model services. Um, so that's, that's the better reason to connect grammar-based sampling and, and open models.

Um, okay. So the problem with fine-tuning, and an annoying thing about prompting, um, is that if there's not a kind of shared... Like the Gorilla model is like fine-tuned on a bunch of APIs from like TorchHub, TensorFlow Hub, and Hugging Face.

So the Gorilla model is really good at using other machine learning models, but not like generic possible tools, at least this example. They m-- They-- Maybe they have tuned more than one.

Um, but this is a, a general problem that if you train a model to use a specific tool, um, then like the, uh, it's not gonna be able to use like any tool.

Um, but if you train a model to, to use a very broad class of tools by using something that's like kind of closer to this grammar where there's like a, a format for tools, um, then you are now a...

People write an interface between the f-- uh, that standard and the, um, and the thing that they actually wanna use. So this has shown up in Open-- uh, like, uh, in OpenAI's API as the, uh, use of JSON schema for describing function calls.

So this allows them to train a model on fairly generic stuff, um, that all fits this... Like it all fits the JSON schema spec. Um, and so the model has learned a bunch of stuff about the JSON schema spec and how to generate that correctly.

Um, you can imagine using grammar-based sampling to enforce that. Um, and this, uh, allows it to connect to many, many tools 'cause now all you need to do is write a tiny connector between like the JSON format and the actual thing you wanna use.

Um, and that's like pretty easy. It's like A big part of web development, from my understanding, is that you just, like, pass JSON blobs back and forth until somebody s- gives you money. [laughs]

Um, and so, uh, yeah, so this is a, a very good kind of schema. Um, uh, but one thing that people miss is that the tool doesn't have to actually be real.

Like, the key thing that happens here is the language model goes from outputting unstructured text to outputting JSON if it's schema, and it just so happens that the primary use case for that that OpenAI envisaged was putting it through a, like, function call, putting it through some downstream computer system.

Um, but, like, really that, uh... Some downstream system. But really, it, like, doesn't have to be a real function. You can tell it about a fake function that's like, "Please pass a string," like, describing whether the, the input was spam or not spam, so that I can, like, render an HTML element, right?

And so the model is now trying to, like, call a function that's like that in order to provide the arguments to that function, it has to decide whether an input is spam or not spam.

And that's maybe the thing you really care about. And so you, like, invent a little fictional function for it to call that it... You don't call and then you just use it for something else.

So this is a pattern in, um, uh... There's a library for this called Instructor from Jason, um, Jason Liu, who's gonna be speaking later at the conference. Yeah?

And is it important to make it act like your app has the function or do you just say, "I want this to be spam"?

Uh, you have to fit the JSON schema, which the schema that they... Like, the sche- the... There's, like, a meta scheme kind of thing. They're like, it has to, it has to be a function call, and the model has been trained on things that are like, you know, get name, get current weather.

Um, so, uh, yeah. I mean, you can hack in... Because, you know, functional programming has taught us that everything is just a function. Like, a constant is just a function that always returns the same thing.

Um, and so you can, you can, like, hack it in there. Um, and Instructor has some fun, like, kind of functional programming stuff built into it, like maybes and, and stuff, so, you know.

Um, and also somebody did, like, DAG construction, where it's like you give it a schema for a DAG constructor, and then it, like, writes a DAG of function calls instead of just a single function call.

So you can really go wild, which is very fun. Um, and yeah?

Most of the time when you generate something, uh, you... If you wanna extract something out of the output you wanna display to the end user, and then the question of latency comes in, that's why you stream it.

Mm. Mm-hmm.

Uh, if we use this, how do we solve the streaming problem?

Yeah. So that is a great question. The answer is that this basically breaks your ability to stream. Um, I think it's not...

So this is maybe a little bit more oriented to, like, back of house stuff, where you're using language models to, like, handle data rather than using language models to directly interact with a user.

Um, I think if you set up a pipeline correctly, then you can stream the outputs from one call into the inputs of the next one. And if you have the relevant information you need from the func- the function call one, then you can just immediately kick off the next thing, and you can just...

You can write, you know, like, more like a Unix pipe style, and then you start to get back to being s- uh, streaming. But you don't have... Like, the Unix pipes work because of new lines as a separator that lets you break work out, and there's not an obvious way to do that with this.

Um, so yeah. The short answer, I guess, is that it's really hard to get back that kind of streaming thing when using these. Um, yeah. Um, I'm gonna... Let's see, how much more do I have?

I'm gonna push forward because I wanna make sure to get to the last section. Um, but I will be around to answer people's questions. Um, okay. So, uh, this conference is not called NLP Engineer Summit, and we've been talking about, like, s- you know, structured out- extracting structured outputs from language, information retrieval, like that's also natural language

processing, and language user interfaces. Like, that's not artificial intelligence. Like, where's the AI? The, like, the thing that really feels like artificial intelligence with language models is something like agents, uh, that are...

That have memory that they keep over time. Uh, so for example, the generative agents, um, that was, uh, let's see. It's mostly Stanford people, if I remember correctly. But, uh, the, like, generative agents paper, uh, combined, like, a stream of memories generated as these agents interacted in, like, a video game environment with, like, some, like,

reasoning flows to create these, like, little tiny characters that had personalities that developed over time in interaction with each other. And, like, um, and that is, uh, like much closer to what people imagine when they hear AI than even a chatbot.

Um, and there's been a lot of advancement in, uh, using these things in simulated environments, so that was, like, a full all language models simulated environment with generative agents.

There's also a ton of really cool stuff going on in the Minecraft world, um, which is, like, people have... Uh, this Voyager agent writes JavaScript code, uh, JavaS- yeah, JavaScript code to call the, like, this, like, Minecraft API that allows it to, like, drive a little, um, uh, you know, a little character in the Minecraft world.

And it starts with basically nothing, um, and then it writes itself a bunch of little subroutines to, like, mine wood log or, like, stab zombie or whatever. And it, like, accumulates them over time, like, learns how to do new stuff, um, like, comes up with its own curriculum for how to sol- like, how to get better, um,

and was able to, like, do extremely well at this, uh, notoriously hard RL task, uh, mine diamond, uh, which was, like, a, a, a grand challenge for the RL world, um, only a couple of years ago.

Um, so So they're, like, they can accumulate information over time, they can accumulate skills over time, they can use tools. This is all very cool. Um, they are... They have a couple of problems, the biggest one being the, like, the problem of reliability.

Um, structured outputs can help with that, and there's, like, only limited work, I would say, on agents that has come out since at least, like, published, you know, research work since the, like, since function calling got really good in the OpenAI API.

Um, th- also, there's kind of, like, a cacophony of different techniques out there with, like, Voyager. Uh, ReAct is kind of an agent, um, generative agents. Um, there's a really s- like, awesome paper from, uh, Tom Griffiths' group at Princeton, Cognitive Architectures for Language Agents, that brings back a bunch of ideas from good old-fashioned AI in the

'80s on, like, um, production systems, uh, and cognitive architectures. A bunch of stuff that was, like, really cool ideas, but it could never, like, get past the demo stage, um, on, like, how to create the things that we know about or that we believe about human and animal cognition, like procedural memories, semantic memories, episodic memories, how to

implement that in software. And the problem... Like, those systems could do cool stuff, but the problem was always that they lacked this, like, general world knowledge and common sense.

With language models, they don't have memory, um, and they don't, um, like, they don't have this, like, structured aspect to their cognition, um, but they do have that, like, world knowledge and that common sense.

Uh, so this is, uh, like mooshing those two things together and using a language model to do, um, basically these kinds of, like, l- uh, ob- observing the world or doing cognition, um, and doing dec- decision procedures, um, like marries the best of both worlds.

Um, and it is actually, like, a pretty effective way of breaking down the existing agent architectures, uh, like, and their different choices about how to do long-term memory, how to do external grounding, how they, like, interact with the external world.

Um, the concept of internal actions, uh, comes from cognitive architectures, um, which is, like, uh, choosing to spend time reasoning or choosing to update your, like, long-term memory, um, or yeah, or your decision procedure.

Yeah, and then also explicitly calling out a decision-making procedure. Um, so there's a t- and that, that paper is also just, like, has an entire research agenda in it, um, on, like, ways that you could just start filling out the cross product, just filling out a big array of, like, try this idea from language models with, like,

this idea from cognitive architectures. Um, and there's just, like, a billion of really cool ideas in there. Um, so if you are interested in agents, um, but have, like, struggled to, like, uh, like, wrap your brain around all the different ways you could, you could do stuff, um, and around, like, how to make them a little bit

more tame, uh, I think the Koala paper has some good pointers.

Um, oh yeah. And then lastly, for this LLM patterns thing, I was talking generally about, like, different ways people are building stuff with LLMs. Eugene Yan's blog has some of the best, uh, writing on this,

um, uh, both on, uh, patterns and anti-patterns.

Okay. Um, I wanna give some time for monitoring evaluation and observability, so I'm just gonna... I know that there's probably lots of interesting things that people have to say on the agent stuff, but we'll, we have the rest of the conference to talk about that.

Um, so, uh, the goal here is to talk about AI engineering. So that last part was about AI. What about the engineering? In engineering, we wanna have a process for building, like, a process for creating these things and a process for improving them.

And progress on this front has been pretty halting. Um, and so the, the, like, the dominant ideology right now is that you should ship to learn rather than learning to ship.

Um, and so this is one of the big ideas in the Full Stack Deep Learning course that I've taught in... It's something that Andrej Karpathy has really hammered on, the, like, idea of a data engine or data flywheel, where in order to do well, you need to go out there and collect data from the world, uh, find

issues in your data, and use that to improve your model in, like, you know, uh, an, an unending cycle. Um, Charity Majors from Honeycomb, uh, who's big in the monitoring and observability world, uh, like, has said that this is something that she's come to like about ML.

In software, you start with tests, and then you graduate to production when the tests pass, or at least, like, that's what you tell people on the internet and, like, your manager.

Um, uh, but with ML, you can't even lie, and you know that you have to, like, start with production and use that to find out the, like, issues with, uh, to, like, generate your tests.

So, you know, it's, it's oops, all regression tests, uh, version. Um, and so what that w- that means is that monitoring is very critical from the very beginning, um, that we monitor for user behavior, we monitor for performance and cost, and we monitor for bugs.

So some of these are just, like, regular old monitoring stuff, and this is just, like, bread and butter things that can be, uh, yeah, like, similar to the way we do with, with, uh, existing software.

Monitoring users always reveals, like, uh, like both misuse and product insights. So one thing that I found from running this Discord bot is, like, one of the things that you get the most are, like, meta questions.

Like, uh, are you getting feedback from these emojis? Who's a good bot? That's maybe not a meta question. Um, does your dataset include your own source code? Um, what do you do?

Like, these are very common, like, things that people input, and it wasn't, it wasn't in my head that that was important. So now there's, like, special stuff in the prompt for handling that class of questions.

So by monitoring how users use the, uh, your, your system, you can get really great product insights. Yeah?

Did you get these from chat?

Oh, yeah. Uh, I had logged them to Gantry, and then I looked at the ones that had up and down thumbs. Um, and I also read all of them 'cause it was, like, only a couple hundred rows.

Oh, man. My battery's gonna run out. Uh, all right, we gotta move fast. Um, so [laughs] uh, monitoring, monitoring performance, uh, can help us manage the constraints that I, like, talked about when we were thinking about all the different places, um, our models might run.

So w- as always, you wanna monitor things like latency quantiles, like, like, how long do requests take? Oh, wow. That's nice. Thank you.

Huge. Um, and so, like, latency quantiles, like, that's how long... Like, take all the requests, what is the probability that a request took at least this long? Um, the...

People often think, like, if I get 90% of them, like, below something, that's great, and the problem with thinking that way is that users don't just make one request, they make many requests in sequence, so by the time you've made, like, 30 requests, there's a 10% chance of hitting, like, a really slow one, then, um, you know,

you have hit a slow request. So that, um, so you really need to care about those, like, 99th percentile latencies. Those are also often your most useful and engaged users.

So watch those, watch those extreme quantiles. Um, and obviously, like, throughput is a distinct thing to also monitor for the quality, uh, you know, quality of the system. Wanna marry that with things like the profiles and traces that I talked about before, like spot check ones randomly subsampled so you can check what, like...

So you can actually debug that throu- the throughput issues. That's fairly general stuff. If you're an inferen- using inference as a service provider, you're gonna wanna monitor API rates and errors, monitor costs.

If you're self-serving inference, you have a lot more stuff to monitor, um, and that's, like, compute utilization. Um, yeah, I guess I already talked about this. Uh, yeah, yeah.

Well, so it's, it's an even hard- Like, maintaining the throughput i- when you're doing the inference yourself is, like, much more your problem, um, and much more, uh, AI, ML-specific stuff.

Um, yeah. Okay. Monitoring for bugs is another can of worms. We'll talk about that in a second. Um, this is, like, just generally this is a very fast-growing field, um, so there are generic monitoring observability solutions for all kinds of, like, you know, complex apps and sh- and, and web apps.

Datadog, Sentry, New Relic, Honeycomb. Like, these are, um... Like, you can adapt those, um, and that might be the thing that wins. Um, there is, uh, you can of course just roll your own with the, like, the, you know, open, open telemetry compliant, uh, you know, tooling, and you could use the existing MLOps tooling, so there's a

lot of stuff that has been built for monitoring observability of general ML applications, so including Weights & Biases where I used to work. Um, Fiddler, Arize, and Gantry are the, like, three larger startups in that space, [smacks lips] um, with more of a focus on monitoring systems in production and less on the, like, MLOps, like, kind of, like, serving,

um, and, like, m- managing, managing training, like Weights & Biases. Um, there's also, because generation times, uh, are now six months or less, a new generation of ops tooling for LLMOps, including LangSmith from LangChain and, uh, LangFuse, which was in Y Combinator's recent batch.

Um, it's, like, the, uh, very unclear which of these is gonna be the, the best solution, so I think it's, like, you know, dealer's choice. Try them all out.

Um, I think I like tools with as much ability to, like, make crazy queries of unstructured data as possible, um, so that's something that I really like about Weights & Biases' production monitoring, uh, offering.

Um, Gantry has some similar stuff. Um, I've tried less of it with the, uh, the other tools. Um, I think if you're doing s- if you're doing it with Datadog, Sentry, et cetera, you're probably gonna need to roll some of that stuff yourself.

Um, but maybe that's fine. Jupyter Notebooks are fun. Um, I was gonna check out, uh, the, like, LangFuse monitoring interface, but, um, in interest of time, gonna go past that.

They have an awesome demo where you can interact with their docs chatbot, and it shows up in their monitoring interface. So, like, they have a live demo of their monitoring tool where you can actually, like, use it to monitor an app that you can also use.

Um, so that's just... It's really, it was really fun to, like, actually try out the, the tool that way. Um, I recommend you try it out. Um, but just monitoring, like, just getting ahold of information is not enough.

This is something that's known from, like, the distributed systems monitoring world. What you really want is observability. What both Charity and Andre were talking about is about how you improve a system based off of what you observe.

It's, like, not enough to just, like, throw something out there and observe... A- and, like, just see the mistakes. You want to, like, fix the mistakes. Um, and so there's this, uh, Honeycomb and, uh, Charity are big on the idea of observability as the, uh, as a, an idea from, like, control theory, from, like, old school, um,

like, control theory, systems theory. Uh, observability is whether you can actually f- um, figure out what is going on inside of a system just from observing it from the outside.

So it's like, can you actually debug this software just from looking at your logs, um, uh, and not having to go into a live debugger inside of the system?

Um, and that's, like, uh, when live debugging does not work and when systems have outpaced our ability to predict what's going to break, um, this is the only solution.

Uh, and for AI systems, that is, um, where we can't predict what's going to break and you can't, like, drop into a debugger 13 layers deep in GPT-3 and...

or GPT-4 and, like, debug, uh, its inference. Uh, you have no choice but to monitor stuff sufficiently that you can fix the issues.

The blocker here is that actually determining whether the model is right or wrong, um, is itself hard, which makes figuring out how to fix it also hard because you don't necessarily know whether it's messing up and you don't know whether you fixed it.

Um, so we're in a tough phase for this problem right now. There will be lots of discussion of evaluation at this conference, which is very exciting. Um, lots of people complaining about how difficult evaluation is, Anthropic and, uh, Arvind Narayanan from, uh, and Sayash Kapoor, who write the AI Snake Oil Substack, really high quality stuff.

Um, and, uh, OpenAI, like open source their eval framework, uh, because in part they like don't... can't really evaluate their system themselves. It's like that's how hard this problem is.

Um, it's also what we saw with the, uh, false promise of imitating proprietary LLMs, like a large community of people were like kind of convinced that models were doing better than they actually were.

Um, so the solution, uh, like is to s- like one of the key solutions is to spend time looking at your data. Stella Biederman from Eleuther has talked about this.

Uh, Jason Wei has talked about how critical this is. Jason is at OpenAI now, um, and talked about spending like a ton of time just like getting very good at evals, like building tooling, internal tooling for evals, spending time with like understanding the evaluations.

Um, and somebody on Hacker News said it's a major differentiator. So, you know, that's, that's definitely the orange website never lies. Um, so, um, evaluation is particularly hard and all these complaints about evaluation are when you're dealing with like open-ended generations from a language model, like no structure to them, um, no s- no real structure to the

user inputs, um, and m- like limited data sources. Um, but there's this nice flowchart, um, from the Full Stack LLM Bootcamp that my fellow instructor Josh Tobin made that sort of helps you avoid getting into that, uh, pit of evaluations.

So if you can find a correct answer, then you can stick with existing ML metrics and you like don't have to worry about the problems of eval-- of like the difficulty of evaluating open-ended generations.

If you have a reference answer, you can check for like reference matching, which is like a looser thing than like a literal correct answer, which is like A, B, C, or D in multiple choice is a correct answer.

A reference answer is like a short like generation, like a short answer on a test. Um, if you have a previous answer from your system, you can at least see if your system is getting better by comparing the two, um, and that like kind of which is better comparison can be done by a human, can be done

by a language model. Um, and if you have human feedback, you can actually check like between, uh, the input and the output was the feedback incorporated by the language model.

Like a human said, "I didn't like that." Um, did the language model get better? And it's only if you don't have any of those things that you like are out in the unstructured world.

Um, the people at Elicit, um, who have worked on doing extraction of information from scientific papers have a very principled approach of iterated decomposition where you start with a task that runs end to end, and then you, uh, when you notice a failure, you look at the failures and you see how you could have broken the task

out into multiple pieces in such a way that the failure would arise in a simpler subtask and then optimize that subtask. So you run into the problem that's been mentioned before about latency if you're like chaining calls, um, and it's not always easy to like decompose the, for example, to decompose the process of responding to a user

in a chatbot. That's kind of challenging. Um, but, uh, but when you can do this, this is another great way to like get yourself out of the hole of needing to evaluate open-ended generations.

But if you're stuck evaluating natural text, there's a couple of like basic approaches. Um, you can just, uh, keep a few trusty test cases at hand, um, and s- uh, you know, if it does well on those couple of test cases, looks good to me, let's ship it.

Um, unclear what to do when it fails, just like hit the language model with a wrench. Um, but, uh, this is what kind of grows out into that data engine.

You start with something like this, and then you start adding stuff from your production observations into it, and then you like put it in a GitHub action and like now that's like, that's, that's basically testing, right?

Um, that's, it's certified software. Um, you can like, uh, you can try and get user feedback. A, you wanna do it as naturally as possible. Um, like, uh, if you're like you...

What you really wanna reveal preferences from user behavior. So the image generation world is very ahead of the language modeling world I think on this, if you look at Midjourney, for example.

Um, that is what Honeycomb did with their query builder. They attached it to, get this, downstream business objectives. Wow, what a way to build a software system. That's the right way to do it.

Um, and so like connecting a chain of metrics from the actual system that you're improving to the actual downstream like organizational goals, um, uh, through things like revealed preferences of users or like, yeah, general user behavior.

Much better than like demanding users fill out a form. Um, you could also pay people to do that work of giving you feedback on your system with an annotation team.

This is what the large, this is what OpenAI does to improve their models. But as alluded to by Karina, it's actually much more effective to use language models in that place because language models are maybe not as smart as all humans, uh, but they tend to outperform crowdworkers, um, on, uh, a large number of very textual tasks.

Um, and so, um, you might find that the task of like annotating and improving your data, if you're at the point where you're starting to think about crowdworkers, you'll find lower cost for equal performance with, uh, like GPT-3.5 Turbo is like a median crowdworker, GPT-4 is like a ninetieth percentile crowdworker.

Um, and maybe a hybrid approach with some crowdworkers, a smaller number of crowdworkers managing, uh, some language models is also been discussed. Um, all right. So the, there's like not that much to say in the end about that, like that aspect of the engineering of systems.

We don't know what the user interfaces and the user experiences are gonna look like. We don't know, uh, we don't know a lot about how to engineer these things to be correct.

Um, so, uh, I guess the exciting thing about that is that the people who are here in this room, on the stream, at this summit are here to fill in all of the steps here that lead to from the circle to the fully drawn owl, um, by like, uh, uh, by figuring it out, um, by like trying

things and sharing what works like people will do at this conference. Um, so that's why I'm excited to be here, and I hope you are as well. All right.

Thank you, everyone. [audience applauding] [upbeat music]