AI Engineer Summit 2025
The LLM Triangle: Engineering Principles for Robust AI Applications
Read the talk
The LLM Triangle: Engineering Reliable Applications
Reliable LLM applications start with an explicit procedure, then match models, software and contextual data to each step of the work.
From a talk by Almog Baku
Before you start: Basic familiarity with LLM prompts and API calls is helpful; the Python example only constructs prompt text.
Why does the proof of concept break?
What separates an impressive LLM prototype from an application that works reliably for customers? Small workflow enhancements and wrappers are an early use of a much broader computing resource. Almog Baku suggests thinking of LLMs alongside CPUs and GPUs: a new resource whose capabilities should influence how we design software. His quantum-computing analogy pushes this toward changing programming languages, but that part is too categorical—Cirq already supports quantum programming in Python. The useful distinction is that LLM applications combine conventional languages such as Python, JavaScript and Go with instructions written in natural language.
That combination makes a proof of concept easy to build and consistent production quality difficult to achieve. The prototype works on the cases its builder has tried; then a real customer supplies the exceptional input that breaks it. The customer sees a failed application, regardless of how impressive the underlying model is. Baku describes LLM-native applications as 10% sophisticated models and 90% experimentation and data-driven engineering, immediately acknowledging that he made up the split. The practical thesis is reliability comes from engineering the application around the model.
Baku grounds the framework in experience founding companies, including one in AI infrastructure, and founding GenAI Israel, which he describes as Israel’s largest such community. At the time of the talk, Baku reports 5,000 engineers, researchers and data scientists in the community. He also reports helping customers build dozens of applications over the preceding two years. The framework distills that application-building experience into a repeatable way to approach the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define the procedure before choosing the implementation
The LLM Triangle connects three choices: the model, the engineering techniques and the contextual data. A fourth element guides all three: the standard operating procedure, or SOP. Borrowed from manufacturing, an SOP specifies how to carry out routine work while maintaining a required level of quality. It gives the application a process to implement rather than merely a desired outcome.
Consider a fulfillment center. A package that comes apart creates support costs, lost goods and reputational damage. The center cannot depend on every packer independently discovering the right method. Instead, it provides a sequence:
- Check that all the items are present.
- Mark them in the system.
- Apply the label.
- Inspect the package.
Baku’s analogy is to treat the LLM as a smart, agreeable intern who is inexperienced at this particular job. Intelligence does not supply knowledge of your organization’s procedure. Give the model an explicit recipe for the task.
For a landing-page builder, writing that recipe starts with an expert interview. Ask how the expert approaches a new page and which elements matter. Then describe the process back to them. Their corrections reveal where your description has skipped something they consider obvious. This decomposition reduces the cognitive load of the task while exposing the decisions the application must reproduce.
A first description might jump directly from gathering information to writing the page. The expert, however, constructs a compelling story between those two steps. That story determines what the landing page should communicate. If the SOP omits it, the implementation asks the model to make an important reasoning jump without guidance. Experts may not notice that they perform these intermediate steps, so extracting them requires specific questions. Once the steps are explicit, draw the process as a graph when loops or recursion make a simple list inadequate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Implement each step with the right tool
Once the SOP describes what to build, engineering techniques provide ways to implement it. Baku groups them by where the work happens:
| Layer | Techniques |
|---|---|
| Prompt | Chain of thought, role assumption, few-shot examples |
| Software | Chains, tools, fine-tuning |
| Across both | ReAct, structured output |
The point of this grouping is to choose a technique according to the step it must accomplish. A prompt technique alone need not carry the entire process.
LLM-native architecture is the arrangement of those steps into a working system. Flow engineering, chains and workflows describe much the same architectural activity here: split the SOP into components, each responsible for a particular transformation. Some components call an LLM; others are ordinary code.
A Wikipedia-style article writer makes the division concrete. Its SOP begins by identifying the subject, researching the literature, writing a table of contents and expanding each section. The implementation adds the necessary intermediate operations:
- Normalize and distill the user’s request.
- Categorize the subject.
- Search Google for relevant information.
- Convert the retrieved HTML into Markdown using code.
- Give the processed material to an LLM component that writes the table of contents.
- Expand the sections.
Baku motivates the HTML conversion with his observation that LLMs work better with Markdown than HTML. The architectural lesson does not require every operation to use a model: use software to prepare the information before asking the model to reason or write with it.
An LLM’s capabilities do not make it the best implementation for every step. Where a transformation can be handled deterministically, conventional code provides consistent behavior and a more direct way to inspect what happened. Reserve model calls for the parts that benefit from them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose where the agent gets discretion
In this framework, an agent is a standalone component of the architecture that calls an LLM: a model plus a prompt template. That is a deliberately broad definition. An agent may be ephemeral, reusable or recursive; its basic responsibility is to take the process from one state to another. An autonomous agent adds discretion over how to accomplish the task, using tools to plan and act.
An assistant can use either design. In the autonomous version, give the model weather and retrieval-augmented generation, or RAG, tools and let it decide how to complete the request. In the crafted version, the application first asks an LLM to classify the message into an explicit enum: emotional support, weather, trip planning or support. That selection determines the next process, such as retrieving knowledge or fetching weather information. A subsequent LLM component composes the response; only that final response is shown to the user.
The autonomous version looks cleaner because the developer writes functions, exposes how to call them and delegates the sequence. It is quick to build and leaves room for creative solutions. The cost appears when behavior varies: an excellent solution on one run may become a failed customer demonstration on another, with a difficult debugging trail. A crafted flow makes the next step explicit, improving predictability and making failures easier to locate, but takes more development time and resources.
| Design | Main advantage | Main cost |
|---|---|---|
| Autonomous tool use | Fast to build; flexible solutions | Variable behavior; harder debugging |
| Crafted flow | Explicit sequence; easier inspection | More implementation effort |
These are trade-offs in how the application controls its work, not a reason to choose one design for every task.
Baku recommends scoping autonomy inside clear boundaries. Give an autonomous component an area in which it can be creative while keeping the surrounding process constrained. If that component produces a poor result, the consequences remain limited to its assigned area. The architecture can therefore combine predictable orchestration with discretion where discretion is useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match the model to the task and its constraints
Different steps need different capabilities. Summarization and entity extraction generally demand less from a model than complex reasoning. Model selection must also account for the environment in which the step runs:
- Infrastructure and performance: Self-hosting an open model brings infrastructure work; proprietary services from Anthropic or OpenAI offer a different operational arrangement. Latency matters alongside capability.
- Cost-effectiveness: Compare the model’s price with the value the task creates and the budget available.
- Data availability: Labeled examples may support few-shot prompting or fine-tuning. A task that instead depends heavily on pretrained knowledge places different demands on the model.
The decision is therefore about a task, its data and its operating constraints—not just a model ranking.
Baku’s model-selection heuristic favors large models for broad capability, with o1 among his examples, while recognizing that capability does not imply low cost or low latency. Smaller models are candidates for simpler tasks, especially when useful task data is available. They are more likely to struggle with complex work; a large model is a stronger starting point for a complex task with a limited dataset.
| Situation | Starting heuristic |
|---|---|
| Simple task with useful examples | Consider a smaller model |
| Complex task with little task data | Start with a larger model |
| Broad capability needed | Expect cost and latency trade-offs |
This is a selection guide, not a measured comparison of the models named in the talk.
The optimization sequence follows from that uncertainty:
- Start with a large model and establish how the application behaves.
- Collect data from the task.
- Add useful few-shot examples, potentially selected dynamically.
- Optimize incrementally, considering fine-tuning or a model change once the accumulated evidence supports it.
Baku places fine-tuning later—potentially after months of learning about the application—rather than making it a prerequisite on day zero. Early model capability buys room to understand the task before taking on that additional work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the input the model actually needs
Data is the triangle’s central practical concern. Baku explains model input through a simplified picture: application software assembles instructions, user messages and context into an input that the model consumes. That picture is useful for thinking about prompt construction, but it is not a portable API contract. The current Claude Messages API, for example, uses user and assistant messages with a separate top-level system parameter; its documentation does not establish the hidden serialization described in the talk.
A prompt template separates reusable instructions from the values supplied for a particular request. Substitute different variables and the application produces a different concrete input while retaining a maintainable template. The template itself does not create memory: a stateless call needs the relevant information included in its input. Organizing data into variables helps the application manage that input; it does not remove the need to decide what the model should receive.
That decision is more demanding than finding one fact in a long document. A needle-in-a-haystack test asks whether the model can recover a hidden item. Baku questions whether success on that test captures application needs: real work may require many relevant facts from a much larger collection. Too much broad context makes the task harder to focus; too little leaves the model without the information needed for a correct answer. Supply enough context to solve the task, and select it for relevance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use examples to specify the intended categories
Consider the instruction to classify a product by its name. Given PlayStation, the model returns gaming console. That is a reasonable answer, but the instruction has not specified the level or vocabulary of classification the application wants. Few-shot examples can communicate that convention through in-context learning, without fine-tuning.
Keep the instruction the same and add TV → consumer good and printer → business equipment. In Baku’s example, the answer for PlayStation changes to consumer electronic. Both answers describe the product, but the second is closer to the intended vocabulary. The examples supply information that the original instruction left ambiguous.
The same prompt construction can be expressed directly in Python. The operation below adds the example pairs while preserving both the instruction and the product being classified:
python
instruction = "Classify the product by its name."
product = "PlayStation"
examples = [
("TV", "consumer good"),
("printer", "business equipment"),
]
def classification_prompt(product, examples):
lines = [instruction]
lines.extend(f"{name} -> {category}" for name, category in examples)
lines.append(f"{product} ->")
return "\n".join(lines)
prompt_without_examples = classification_prompt(product, [])
prompt_with_examples = classification_prompt(product, examples)
These strings are inputs for a model call; the code does not itself assign a category. The teaching mechanism is to show the desired convention through examples when describing it in words is difficult. Baku compresses that principle into “show, don’t tell.”
Adding examples without changing the request
Constructed example: Arrow formatting and object IDs are teaching representations matching the adjacent Python code. The comparison shows prompt construction, not an executed model call.
PlayStation
Operation: Insert the two demonstration pairs into the classification prompt while preserving the instruction and product.
Instruction
Classify the product by its name.
Classify the product by its name.
TV example
Not present
TV -> consumer good
Printer example
Not present
printer -> business equipment
Classification request
PlayStation ->
PlayStation ->
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Select examples for the current request
A large example collection does not need to become one large prompt. Dynamic few-shot selection chooses examples for the current use case. Baku suggests selecting five or seven examples rather than supplying the entire collection. Similarity-based selection can use vector search, much like RAG; rule-based selection can use keywords or categories. In either case, the application retrieves demonstrations of how to perform the task, choosing the subset most useful for this request.
This returns the triangle to its organizing relationship: the model, engineering techniques and contextual data all operate under the SOP. The procedure specifies the work; engineering techniques implement and specialize it; the data gives the model what it needs at each step.
Baku closes by putting data first. Inspect the data available to the application, then use the SOP and engineering techniques to shape what reaches the model. Better task-specific input is the practical route toward better output. He ends by inviting questions through LinkedIn and pointing viewers to the full white paper; his website lists his publications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original paper on interleaving reasoning with actions that interact with external information and environments.
Further reading
Baku's earlier presentation explains SOPs, prompt techniques and relevant context, with a public transcript and discussion.
A long-context benchmark covering multiple-needle retrieval, multi-hop tracing and aggregation beyond simple needle-in-a-haystack tests.
Updates since the talk
Current request structure for Claude messages, system instructions and stateless conversations.
Read the complete timestamped transcript
- 0:00
Hey, thank you for having me, and thank you for being here today. Here, we're going to talk about the LLM Triangle Principles, engineering principles for building robust AI applications.
- 0:14
My name is Almog, and I'm gonna talk with you about, surprise, surprise, LLMs. So this is a prerecorded session, but I think it's a safe bet to say that everyone, everyone told you today or gonna to-- tell you today that LLMs are amazing.
- 0:33
They're gonna change the world. They, they basically gonna shift the way how we think about technology and software.
- 0:41
And to be honest, most of the use cases today are just like sparkers, like small projects that a little bit enhance the workflow.
- 0:55
And regarding to what they said, I totally agree with them. LLMs are amazing. They're blowing my mind.
- 1:02
And today, this is just the beginning. This, this is just the tip of the iceberg.
- 1:09
Most of the use cases today might be LLM wrappers or GPT wrappers, but the future
- 1:17
gonna blow your mind. But in order to get there, we need to shift our thinking. Uh, I think that a good analogy for that is thinking of LLMs like a new resource, like GPU or CPU, and Google just showed us today that quantum computer is around the corner.
- 1:36
Um, actually, it wasn't today. It was a few weeks ago, but never mind. Um,
- 1:41
but everyone know that when quantum computer will be here, nobody will write Python to program quantum computing applications. So why do we do that with LLMs? Um, I think this is very confusing, uh, because we do build LLM applications with Python or, I don't know, resembling JavaScript or
- 2:06
PHP. Just kidding. Amazing languages. Not really. Yes, they do. Um,
- 2:14
y- we do write them with Python and JavaScript and Go and whatever, but we also write them in native language. And I guess that everyone here that tried to build something or built something know that building POCs is very easy.
- 2:35
But production applications that's performing in a certain quality, that's damn hard. Because every time we take the POC and show that in our demo to a real customer, every time, we just facing the only use case that makes everything break, and the customer is just d- don't get it.
- 2:58
Like, "Why the hell are you wasting my time?" So
- 3:03
over the past few month, um, past few re- years, I've been thinking about it, and I, I came to the realization that LLM-native applications are ten percent sophisticated models and ninety percent of hard experimentation and data-driven engineering work.
- 3:22
And trust me about that, I'm an engineer. I just made this up. Uh, uh, so, uh, let's wrap it up. So my name is Almog. I'm a serial entrepreneur.
- 3:33
I started a few companies. My previous company was in the AI infrastructure world. I also the founder of GenAI Israel community. Uh, we are the largest community in Israel.
- 3:45
We have five thousands engineers and researchers and data scientists working on GenAI applications. And over the past two years, I've been helping to build dozens of applications, uh, for my customers.
- 4:00
Um, and over the next few minutes, I'm gonna help you, uh,
- 4:08
to maybe, uh, uh, see the way I see about applications and learn from my experience. So I'm trying to distill my experience into, I don't know, the next fifteen, twenty-ish minutes, and to help you to reproduce what I did.
- 4:28
All right. So the LLM Triangle Principles build upon a few principles. Uh, we have the model, obviously, the foundation model, the engineering techniques, and the data. And all of that are guided by, of course, the fourth principle of the triangle, which is the standard operation procedure.
- 4:55
If you haven't heard about standard operating procedure, that's totally fine. That probably meant that you haven't worked in a huge corporation or a factory. So to be fair, I borrowed this terminology from the info-- uh, uh, manufacturing industry, and it basically means that standard operating procedure is a set of step-by-step instructions compiled by
- 5:19
loud, by large organizations to help you, uh, to help the worker to carry out a routine work while maintaining a certain quality. So let's think about a huge fulfillment center, right?
- 5:34
They can't allow themselves to hire some PhD in physics in order to wrap some package, so it won't, I don't know, get lost or, or, uh, get unwrapped, right?
- 5:47
Because if a package gets unwrapped, that cost them a lot of money, like customer support, the, the price of goods, their name, etc. So instead of that, they just hire a lot of interns and tell them exactly what you do-- what to do.
- 6:04
First, you need to check all the items are there, and you need to mark them in our system. And then you put the label, and then you make sure the, the package is all right or whatever.
- 6:17
That's exactly how we should look of the LLM, right? We should look of the LLM like an intern. He's really smart.
- 6:28
He's really, he's really nice guy, right? He, he says, "Yeah, you're right," about whatever we say.
- 6:35
And on the other hand, he's inexperienced on our ta-- at our task. So we should prompt it and tell it exactly what to do, if you will, like a step-by-step recipe.
- 6:48
This way, we can ensure it's excelling our task. So how do we do that? Uh, in order to, to think about that, let's take an example. So our example gonna be how to build a landing page builder.
- 7:03
So first, we want to assume that w-we are the expert, or maybe to interview a real expert, and to reduce the cognitive load of the task by simplifying it into some smaller steps.
- 7:18
So we'll start to interview the expert, or assume to be the expert. How would you f- approach this task? How would you create a new landing page? What's the key elements you consider?
- 7:30
And then we will reflect on it, and we'll phrase it for him. Usually, the expert will say, "Oh, no, no, that's not what I'm doing. I'm doing this and this and this."
- 7:40
And this will allow us to have more information about the nuances that the expert know, but n-not always tell us, right? So eventually we'll have, like, a list of steps, and our goal is to find the hidden implicit, uh, cognition jumps that the expert know, but we don't.
- 8:02
Because it's probably obvious that after you gather data, you write the page, but in between
- 8:11
the expert writing a compelling story that the landing page should deliver to us. And that might be obvious to him, but not for us. And this expert, she might, she might know all of the nuances because obviously everybody know them, but we don't, and we need to identify them.
- 8:35
And sometimes they will do that, and they won't even notice that they do that. So we need to be very specific and ask very good questions, or at least assume to be them.
- 8:47
Uh, all right. After that, we'll compile a list, and I found that sometimes it's very helpful to describe this process as a graph because some of the processes are like, like loop or like we have a recursion there.
- 9:03
So we-- It's very helpful to draw it. All right. So now we know what to build. Now that's, that's the time to actually build it. So the engineering techniques, that's exactly what they do.
- 9:16
It's a set of tools we can use in order to implement our SOP. So there are basically two types of, uh, engineering techniques. We have engineering techniques solely in the prompt layer, like chain of thought, role assumption, few shots, et cetera.
- 9:33
And we have some techniques that are solely in the software, right? Like chains, tools, fine-tuning. And obviously, we have everything that in between, like ReAct, structured output. And we need to try and think about what do we want to achieve and how do we achieve it.
- 9:51
I want to focus today on two critical techniques. The first one is the LLM-native architecture. All right. To simply put, it's just an architecture. All right? Some people call it flow engineering or chains or workflow.
- 10:10
It's all the same. It's simply an architecture.
- 10:15
What we do here is we take the SOP, and we split based on the SOP, the process into multiple steps, multiple agents, if you will. Every agent will do something different, and sometimes we'll have some piece of code that will do that.
- 10:33
So for example, let's say we are building a Wikipedia writer. So the SOP said that we start to understand what the real subject we want to write about, and then we do some literature with researching it, and then we write a table of content, and then we expand on each section, right?
- 10:50
So in order to build that, we'll split it into some process. So first, we normalize and distill the user request. We categorize the subject. We do some Google search and find some data.
- 11:03
And if you haven't noticed, LLMs are really good with Markdown and they are so-so with HTML. So before we're gonna deliver this information to the LLM, we're gonna, uh, uh, we're gonna process it with some piece of code that format it to a Markdown and then to another agent that write the table of contents and et cetera.
- 11:25
So you can notice that some parts are done by LLM and some parts are we process it using a regular software. So LLM is an amazing piece of technology.
- 11:40
Seriously, it's mind-blowing. That being said, it doesn't mean that we need to do everything with LLMs, right? Always [laughs]
- 11:49
LLMs, like, like a, a deterministic code will always be deterministic. It will always provide us with consistent quality. So you might notice that I said agents, agents, agents. What are agents?
- 12:06
So based on the LLM Triangle Principles, agent is simply an LLM. Uh, LLM agent is a standalone component of the LLM architecture that involve calling an LLM Simply put it, it's LLM model plus prompt equal agent, and by prompt I mean prompt template.
- 12:28
And not all LLM agents are equal. Some of them are inferral, some of them are reusable, some of them are recursive, and basically, uh, an agent is just take me from, from A to B by doing something.
- 12:48
And we have a special kind of el-- of agent, which is the autonomous agent. Autonomous agent are given with the liberty, with the right to decide how to implement this task.
- 13:05
So it's basically agent with some tools. So the agent can decide how to plan and how to solve the problem. So take me from A to B by doing something.
- 13:20
I don't care how, just do it. And we can implement the same problem with both ways, right? So for example, let's say we just want to build some assistant, right?
- 13:34
So we can say, "Hey, you are a smart assistant with access to the weather and RAG tool," uh, the, the, the knowledge tool, right? And the agent will just do whatever it wants in order to complete the task.
- 13:49
Or we can achieve exactly the same with an architecture, right? First, we'll process the, the, the re-- the, the message. What does the user want from us? Emotional support, to know the weather, to plan some trip, to s- to, to get some support, and we process that with some agent, uh, and then we get like, like simply
- 14:12
to... We feed the agent with enum and tell the agent, "Hey, just select one of them," and then based on each selection, we'll do different processes. One of them will fetch information from RAG, the other one maybe will fetch some weather information, and then the result of that will deliver to a thir- a, a second agent, which
- 14:33
composing the response, and only that the user gonna see.
- 14:38
And that's pretty complex. Why would we want that? Then like the, the first option is much simpler, it's much cleaner. Like life is full of trade-offs, right? There's no such magic.
- 14:52
So autonomous agents are elegant software, seriously amazing, very creative, very fast to build, to build. Like you simply write some functions and provide the LLM with the, the way to call them.
- 15:10
That being said, they are very hard to debug, and their quality might be unsustainable. Like one time you might have a very creative, amazing solution, and the other time, exactly at the point when you demoing to the customer, it's gonna fail.
- 15:26
In production, we need to have some sort of certainty, and with a crafted flow, uh, we have a sustainable quality. Uh, the process is predictable. We know what's going after each step.
- 15:42
Higher possibility to debug it, and from the other side, it's very expensive because we need to invest a lot of time and a lot of resources to handcraft and develop it.
- 15:57
My two cents is that reality, like always, depends. The way I see it is that I highly recommend to scope, uh, the autonomous agents. So, so like they can be amazing as long as they have clear borders.
- 16:16
Like this is the way-- This is the area that you can be creative. Um, so worst case, it won't deliver you good results, but it's very hedged.
- 16:29
All right, so we've discussed the engineering techniques. Now let's talk about the model. Not all LLMs created equal. We need to match the task
- 16:43
to the model. There are various factors to select the model. The complexity of the task. Uh, for example, simpler tasks like summarization or entity exa- extractions are easier for LLMs to fulfill rather than having a complex reasoning about some,
- 17:05
some thought they need to compile. Um, infrastructure and performance. Uh, a lot of models are out there open sourced and require us to build some sort of infrastructure in order to run them versus some proprietary software, which is more expensive, and we can use like the large models from
- 17:29
Anthropic or OpenAI. Also, we have the consideration of latency, right? Uh, the cost effectiveness. Uh, what's the price of the model versus how much it worth for us?
- 17:46
Um, wha- what's our budget constraints? Uh, the data available for us. If we have a lot of labeled data, maybe we can do some fine-tuning, or maybe we can feed it with some few shots.
- 18:00
But if we rely heavily on the model knowledge, then we need probably a different model. Uh, I found that this chart very helps me. Um, so all right, let's go through it very quick.
- 18:15
Large model are expensive, and they're probably good at most use cases, right? We can throw most problems on o1 or o3 or o9, and it will be amazing.
- 18:29
It doesn't mean that it will be fast and cheap.
- 18:33
Smaller models are pretty fine with simpler task or-- and when we have a lot of data to feed them.
- 18:43
Um, smaller model, uh, with a very complex task probably will underperform, and large models with complex task and limited data set will probably perform fine.
- 19:01
So what should we do? My two cent is
- 19:06
you gonna build an LLM-native application. It's not a short rodeo. It's gonna take you time. Start with the big model, collect data, understand what's going on, and then optimize it incrementally.
- 19:18
In-incrementally, sorry. Uh, that means that you start with a big model, you understand what's going on, maybe you leverage some few-shot you creating, maybe some dynamic few-shot, we'll talk about it later.
- 19:33
And maybe after a few month, you'll fine-tune or change the model. We don't need to, to, to jump into the, the heavy lifting of fine-tuning from day zero, right?
- 19:46
All right. We can't talk about LLMs without talking about data. It's all about the data. Seriously. You can forget everything and just focus on the data because this is the mot-- most important part of my talk.
- 20:03
So it's important to realize that the model doesn't really understand what context or data is, or user prompt, or even system prompt. It only get a s-- It only gets a single text, a prompt.
- 20:20
So when we use, um, OpenAI or Claude APIs, we provide it with a system prompt and a user prompt and another system prompt. Behind the scene, uh, there is a piece of software that takes all of these pieces, glue them together with some template and provide it to the model.
- 20:40
The model sees some input text and return you with some output text. That's the way it work.
- 20:47
And we [chuckles] as builders, we need to make sense out of it. So we split it. And a very important piece of concept is the prompt template. Basically,
- 21:02
um, we have some variables. We can place some variables in our template and to feed it to the model. So we have like a prompt template, but each time it's slightly different.
- 21:16
Again, the model doesn't have any memory. It doesn't understand what the context and the variables are. It's stateless. It's just a trick for us, but it help us to maintain some piece of sustainable code, right?
- 21:32
And the key is to make the data relevant, right? Maybe you've heard about the needle in the haystack problem. So like the needle in the haystack problem says that if we'll feed the model with tons of data, and we'll place some needle inside of the haystack,
- 21:52
maybe the model won't find it. And recent, uh, publications, uh, by the LLM vendors said, "Oh, now we solved the problem. Now we can find this piece of hidden needle in our text."
- 22:07
To be honest, I don't really believe them because in reality, we don't really have a needle in a haystack. We have a lot of hindle-- needles in a huge haystack.
- 22:19
So if we'll feed the model with a lot of data, and we'll ask the model to understand something, that would be very hard. From the other side, if we won't provide the model with enough information, obviously it won't be able to provide us with the right answer.
- 22:36
So we need to find the right balance, uh, of giving the model the context, but not too much a broader context, and how to focus the data we provide it.
- 22:51
Um, there are a few practices and techniques how to teach the model what to do. Um, for example, few-shots. So we can ask the model, "Hey, uh, please classify the product by its names."
- 23:08
And we say PlayStation, and the model will return us gaming console. That's right. But we meant something slightly different. By providing the model with some examples, we can basically utilize one of the greatest, uh, uh, qualities of these foundation models, their way to be in-context learners, to
- 23:33
learn something from the context we provide without fine-tuning it. So classify the product by its name, the same system prompt. But when I say TV, you say consumer good.
- 23:45
TV, consumer good, and we just play with it, right? When I say printer, you say business equipment. Printer, business equipment. When I say PlayStation, you say... And then the model suddenly use a very different language.
- 23:59
It said consumer electronic. All right, so both of the answers are correct, but the second answer is more in the scent. It's more in the vibe we are looking for, which is consumer electronic, right?
- 24:14
So that's a very good way to teach the model some concept that is very, very hard to, to describe in words, right? So we teach it by example.
- 24:27
I think that a, a, a key phrase to remember is show, don't tell.
- 24:34
Sometimes it's easier to teach by showing and not by explaining.
- 24:39
That's one of the key moments to use few-shots. Some companies took this concept even further. They said, "All right, we have tons of examples, and based off the use case, we're gonna find you the best example for w- for the model."
- 24:55
And based of the-- based on the use case, we're only selecting a handful set of examples. Not all the examples, only five or seven. Um, we can use some similarity mechanisms like vectoric search or something like that, much like RAG, or we can do some logic selection of the, of the, of the, of the example,
- 25:21
like by keyword or by category or whatever. Um, so this is the LLM Triangle. It, uh, composed, uh, by three, uh, key, uh, uh, principles, the model, the engineering techniques, and the contextual data, all guided by the SOP.
- 25:44
Again, data is the king. Without data, we don't have a kingdom.
- 25:51
Look at your data, try to think how you specialize it using the engineering techniques, the SOP,
- 25:59
to feed it to the model and to yield higher quality.
- 26:04
Thank you. Uh, if you have any question, please reach out to me over LinkedIn. Um, you can read the full white paper using this link, and thank you. Have a good one.