The LLM Triangle: Engineering Principles for Robust AI Applications

Talk transcript

1. 0:00

   Hey, thank you for having me, and thank you for being here today. Here, we're going to talk about the LLM Triangle Principles, engineering principles for building robust AI applications.

2. 0:14

   My name is Almog, and I'm gonna talk with you about, surprise, surprise, LLMs. So this is a prerecorded session, but I think it's a safe bet to say that everyone, everyone told you today or gonna to-- tell you today that LLMs are amazing.

3. 0:33

   They're gonna change the world. They, they basically gonna shift the way how we think about technology and software.

4. 0:41

   And to be honest, most of the use cases today are just like sparkers, like small projects that a little bit enhance the workflow.

5. 0:55

   And regarding to what they said, I totally agree with them. LLMs are amazing. They're blowing my mind.

6. 1:02

   And today, this is just the beginning. This, this is just the tip of the iceberg.

7. 1:09

   Most of the use cases today might be LLM wrappers or GPT wrappers, but the future

8. 1:17

   gonna blow your mind. But in order to get there, we need to shift our thinking. Uh, I think that a good analogy for that is thinking of LLMs like a new resource, like GPU or CPU, and Google just showed us today that quantum computer is around the corner.

9. 1:36

   Um, actually, it wasn't today. It was a few weeks ago, but never mind. Um,

10. 1:41

    but everyone know that when quantum computer will be here, nobody will write Python to program quantum computing applications. So why do we do that with LLMs? Um, I think this is very confusing, uh, because we do build LLM applications with Python or, I don't know, resembling JavaScript or

11. 2:06

    PHP. Just kidding. Amazing languages. Not really. Yes, they do. Um,

12. 2:14

    y- we do write them with Python and JavaScript and Go and whatever, but we also write them in native language. And I guess that everyone here that tried to build something or built something know that building POCs is very easy.

13. 2:35

    But production applications that's performing in a certain quality, that's damn hard. Because every time we take the POC and show that in our demo to a real customer, every time, we just facing the only use case that makes everything break, and the customer is just d- don't get it.

14. 2:58

    Like, "Why the hell are you wasting my time?" So

15. 3:03

    over the past few month, um, past few re- years, I've been thinking about it, and I, I came to the realization that LLM-native applications are ten percent sophisticated models and ninety percent of hard experimentation and data-driven engineering work.

16. 3:22

    And trust me about that, I'm an engineer. I just made this up. Uh, uh, so, uh, let's wrap it up. So my name is Almog. I'm a serial entrepreneur.

17. 3:33

    I started a few companies. My previous company was in the AI infrastructure world. I also the founder of GenAI Israel community. Uh, we are the largest community in Israel.

18. 3:45

    We have five thousands engineers and researchers and data scientists working on GenAI applications. And over the past two years, I've been helping to build dozens of applications, uh, for my customers.

19. 4:00

    Um, and over the next few minutes, I'm gonna help you, uh,

20. 4:08

    to maybe, uh, uh, see the way I see about applications and learn from my experience. So I'm trying to distill my experience into, I don't know, the next fifteen, twenty-ish minutes, and to help you to reproduce what I did.

21. 4:28

    All right. So the LLM Triangle Principles build upon a few principles. Uh, we have the model, obviously, the foundation model, the engineering techniques, and the data. And all of that are guided by, of course, the fourth principle of the triangle, which is the standard operation procedure.

22. 4:55

    If you haven't heard about standard operating procedure, that's totally fine. That probably meant that you haven't worked in a huge corporation or a factory. So to be fair, I borrowed this terminology from the info-- uh, uh, manufacturing industry, and it basically means that standard operating procedure is a set of step-by-step instructions compiled by

23. 5:19

    loud, by large organizations to help you, uh, to help the worker to carry out a routine work while maintaining a certain quality. So let's think about a huge fulfillment center, right?

24. 5:34

    They can't allow themselves to hire some PhD in physics in order to wrap some package, so it won't, I don't know, get lost or, or, uh, get unwrapped, right?

25. 5:47

    Because if a package gets unwrapped, that cost them a lot of money, like customer support, the, the price of goods, their name, etc. So instead of that, they just hire a lot of interns and tell them exactly what you do-- what to do.

26. 6:04

    First, you need to check all the items are there, and you need to mark them in our system. And then you put the label, and then you make sure the, the package is all right or whatever.

27. 6:17

    That's exactly how we should look of the LLM, right? We should look of the LLM like an intern. He's really smart.

28. 6:28

    He's really, he's really nice guy, right? He, he says, "Yeah, you're right," about whatever we say.

29. 6:35

    And on the other hand, he's inexperienced on our ta-- at our task. So we should prompt it and tell it exactly what to do, if you will, like a step-by-step recipe.

30. 6:48

    This way, we can ensure it's excelling our task. So how do we do that? Uh, in order to, to think about that, let's take an example. So our example gonna be how to build a landing page builder.

31. 7:03

    So first, we want to assume that w-we are the expert, or maybe to interview a real expert, and to reduce the cognitive load of the task by simplifying it into some smaller steps.

32. 7:18

    So we'll start to interview the expert, or assume to be the expert. How would you f- approach this task? How would you create a new landing page? What's the key elements you consider?

33. 7:30

    And then we will reflect on it, and we'll phrase it for him. Usually, the expert will say, "Oh, no, no, that's not what I'm doing. I'm doing this and this and this."

34. 7:40

    And this will allow us to have more information about the nuances that the expert know, but n-not always tell us, right? So eventually we'll have, like, a list of steps, and our goal is to find the hidden implicit, uh, cognition jumps that the expert know, but we don't.

35. 8:02

    Because it's probably obvious that after you gather data, you write the page, but in between

36. 8:11

    the expert writing a compelling story that the landing page should deliver to us. And that might be obvious to him, but not for us. And this expert, she might, she might know all of the nuances because obviously everybody know them, but we don't, and we need to identify them.

37. 8:35

    And sometimes they will do that, and they won't even notice that they do that. So we need to be very specific and ask very good questions, or at least assume to be them.

38. 8:47

    Uh, all right. After that, we'll compile a list, and I found that sometimes it's very helpful to describe this process as a graph because some of the processes are like, like loop or like we have a recursion there.

39. 9:03

    So we-- It's very helpful to draw it. All right. So now we know what to build. Now that's, that's the time to actually build it. So the engineering techniques, that's exactly what they do.

40. 9:16

    It's a set of tools we can use in order to implement our SOP. So there are basically two types of, uh, engineering techniques. We have engineering techniques solely in the prompt layer, like chain of thought, role assumption, few shots, et cetera.

41. 9:33

    And we have some techniques that are solely in the software, right? Like chains, tools, fine-tuning. And obviously, we have everything that in between, like ReAct, structured output. And we need to try and think about what do we want to achieve and how do we achieve it.

42. 9:51

    I want to focus today on two critical techniques. The first one is the LLM-native architecture. All right. To simply put, it's just an architecture. All right? Some people call it flow engineering or chains or workflow.

43. 10:10

    It's all the same. It's simply an architecture.

44. 10:15

    What we do here is we take the SOP, and we split based on the SOP, the process into multiple steps, multiple agents, if you will. Every agent will do something different, and sometimes we'll have some piece of code that will do that.

45. 10:33

    So for example, let's say we are building a Wikipedia writer. So the SOP said that we start to understand what the real subject we want to write about, and then we do some literature with researching it, and then we write a table of content, and then we expand on each section, right?

46. 10:50

    So in order to build that, we'll split it into some process. So first, we normalize and distill the user request. We categorize the subject. We do some Google search and find some data.

47. 11:03

    And if you haven't noticed, LLMs are really good with Markdown and they are so-so with HTML. So before we're gonna deliver this information to the LLM, we're gonna, uh, uh, we're gonna process it with some piece of code that format it to a Markdown and then to another agent that write the table of contents and et cetera.

48. 11:25

    So you can notice that some parts are done by LLM and some parts are we process it using a regular software. So LLM is an amazing piece of technology.

49. 11:40

    Seriously, it's mind-blowing. That being said, it doesn't mean that we need to do everything with LLMs, right? Always [laughs]

50. 11:49

    LLMs, like, like a, a deterministic code will always be deterministic. It will always provide us with consistent quality. So you might notice that I said agents, agents, agents. What are agents?

51. 12:06

    So based on the LLM Triangle Principles, agent is simply an LLM. Uh, LLM agent is a standalone component of the LLM architecture that involve calling an LLM Simply put it, it's LLM model plus prompt equal agent, and by prompt I mean prompt template.

52. 12:28

    And not all LLM agents are equal. Some of them are inferral, some of them are reusable, some of them are recursive, and basically, uh, an agent is just take me from, from A to B by doing something.

53. 12:48

    And we have a special kind of el-- of agent, which is the autonomous agent. Autonomous agent are given with the liberty, with the right to decide how to implement this task.

54. 13:05

    So it's basically agent with some tools. So the agent can decide how to plan and how to solve the problem. So take me from A to B by doing something.

55. 13:20

    I don't care how, just do it. And we can implement the same problem with both ways, right? So for example, let's say we just want to build some assistant, right?

56. 13:34

    So we can say, "Hey, you are a smart assistant with access to the weather and RAG tool," uh, the, the, the knowledge tool, right? And the agent will just do whatever it wants in order to complete the task.

57. 13:49

    Or we can achieve exactly the same with an architecture, right? First, we'll process the, the, the re-- the, the message. What does the user want from us? Emotional support, to know the weather, to plan some trip, to s- to, to get some support, and we process that with some agent, uh, and then we get like, like simply

58. 14:12

    to... We feed the agent with enum and tell the agent, "Hey, just select one of them," and then based on each selection, we'll do different processes. One of them will fetch information from RAG, the other one maybe will fetch some weather information, and then the result of that will deliver to a thir- a, a second agent, which

59. 14:33

    composing the response, and only that the user gonna see.

60. 14:38

    And that's pretty complex. Why would we want that? Then like the, the first option is much simpler, it's much cleaner. Like life is full of trade-offs, right? There's no such magic.

61. 14:52

    So autonomous agents are elegant software, seriously amazing, very creative, very fast to build, to build. Like you simply write some functions and provide the LLM with the, the way to call them.

62. 15:10

    That being said, they are very hard to debug, and their quality might be unsustainable. Like one time you might have a very creative, amazing solution, and the other time, exactly at the point when you demoing to the customer, it's gonna fail.

63. 15:26

    In production, we need to have some sort of certainty, and with a crafted flow, uh, we have a sustainable quality. Uh, the process is predictable. We know what's going after each step.

64. 15:42

    Higher possibility to debug it, and from the other side, it's very expensive because we need to invest a lot of time and a lot of resources to handcraft and develop it.

65. 15:57

    My two cents is that reality, like always, depends. The way I see it is that I highly recommend to scope, uh, the autonomous agents. So, so like they can be amazing as long as they have clear borders.

66. 16:16

    Like this is the way-- This is the area that you can be creative. Um, so worst case, it won't deliver you good results, but it's very hedged.

67. 16:29

    All right, so we've discussed the engineering techniques. Now let's talk about the model. Not all LLMs created equal. We need to match the task

68. 16:43

    to the model. There are various factors to select the model. The complexity of the task. Uh, for example, simpler tasks like summarization or entity exa- extractions are easier for LLMs to fulfill rather than having a complex reasoning about some,

69. 17:05

    some thought they need to compile. Um, infrastructure and performance. Uh, a lot of models are out there open sourced and require us to build some sort of infrastructure in order to run them versus some proprietary software, which is more expensive, and we can use like the large models from

70. 17:29

    Anthropic or OpenAI. Also, we have the consideration of latency, right? Uh, the cost effectiveness. Uh, what's the price of the model versus how much it worth for us?

71. 17:46

    Um, wha- what's our budget constraints? Uh, the data available for us. If we have a lot of labeled data, maybe we can do some fine-tuning, or maybe we can feed it with some few shots.

72. 18:00

    But if we rely heavily on the model knowledge, then we need probably a different model. Uh, I found that this chart very helps me. Um, so all right, let's go through it very quick.

73. 18:15

    Large model are expensive, and they're probably good at most use cases, right? We can throw most problems on o1 or o3 or o9, and it will be amazing.

74. 18:29

    It doesn't mean that it will be fast and cheap.

75. 18:33

    Smaller models are pretty fine with simpler task or-- and when we have a lot of data to feed them.

76. 18:43

    Um, smaller model, uh, with a very complex task probably will underperform, and large models with complex task and limited data set will probably perform fine.

77. 19:01

    So what should we do? My two cent is

78. 19:06

    you gonna build an LLM-native application. It's not a short rodeo. It's gonna take you time. Start with the big model, collect data, understand what's going on, and then optimize it incrementally.

79. 19:18

    In-incrementally, sorry. Uh, that means that you start with a big model, you understand what's going on, maybe you leverage some few-shot you creating, maybe some dynamic few-shot, we'll talk about it later.

80. 19:33

    And maybe after a few month, you'll fine-tune or change the model. We don't need to, to, to jump into the, the heavy lifting of fine-tuning from day zero, right?

81. 19:46

    All right. We can't talk about LLMs without talking about data. It's all about the data. Seriously. You can forget everything and just focus on the data because this is the mot-- most important part of my talk.

82. 20:03

    So it's important to realize that the model doesn't really understand what context or data is, or user prompt, or even system prompt. It only get a s-- It only gets a single text, a prompt.

83. 20:20

    So when we use, um, OpenAI or Claude APIs, we provide it with a system prompt and a user prompt and another system prompt. Behind the scene, uh, there is a piece of software that takes all of these pieces, glue them together with some template and provide it to the model.

84. 20:40

    The model sees some input text and return you with some output text. That's the way it work.

85. 20:47

    And we [chuckles] as builders, we need to make sense out of it. So we split it. And a very important piece of concept is the prompt template. Basically,

86. 21:02

    um, we have some variables. We can place some variables in our template and to feed it to the model. So we have like a prompt template, but each time it's slightly different.

87. 21:16

    Again, the model doesn't have any memory. It doesn't understand what the context and the variables are. It's stateless. It's just a trick for us, but it help us to maintain some piece of sustainable code, right?

88. 21:32

    And the key is to make the data relevant, right? Maybe you've heard about the needle in the haystack problem. So like the needle in the haystack problem says that if we'll feed the model with tons of data, and we'll place some needle inside of the haystack,

89. 21:52

    maybe the model won't find it. And recent, uh, publications, uh, by the LLM vendors said, "Oh, now we solved the problem. Now we can find this piece of hidden needle in our text."

90. 22:07

    To be honest, I don't really believe them because in reality, we don't really have a needle in a haystack. We have a lot of hindle-- needles in a huge haystack.

91. 22:19

    So if we'll feed the model with a lot of data, and we'll ask the model to understand something, that would be very hard. From the other side, if we won't provide the model with enough information, obviously it won't be able to provide us with the right answer.

92. 22:36

    So we need to find the right balance, uh, of giving the model the context, but not too much a broader context, and how to focus the data we provide it.

93. 22:51

    Um, there are a few practices and techniques how to teach the model what to do. Um, for example, few-shots. So we can ask the model, "Hey, uh, please classify the product by its names."

94. 23:08

    And we say PlayStation, and the model will return us gaming console. That's right. But we meant something slightly different. By providing the model with some examples, we can basically utilize one of the greatest, uh, uh, qualities of these foundation models, their way to be in-context learners, to

95. 23:33

    learn something from the context we provide without fine-tuning it. So classify the product by its name, the same system prompt. But when I say TV, you say consumer good.

96. 23:45

    TV, consumer good, and we just play with it, right? When I say printer, you say business equipment. Printer, business equipment. When I say PlayStation, you say... And then the model suddenly use a very different language.

97. 23:59

    It said consumer electronic. All right, so both of the answers are correct, but the second answer is more in the scent. It's more in the vibe we are looking for, which is consumer electronic, right?

98. 24:14

    So that's a very good way to teach the model some concept that is very, very hard to, to describe in words, right? So we teach it by example.

99. 24:27

    I think that a, a, a key phrase to remember is show, don't tell.

100. 24:34

     Sometimes it's easier to teach by showing and not by explaining.

101. 24:39

     That's one of the key moments to use few-shots. Some companies took this concept even further. They said, "All right, we have tons of examples, and based off the use case, we're gonna find you the best example for w- for the model."

102. 24:55

     And based of the-- based on the use case, we're only selecting a handful set of examples. Not all the examples, only five or seven. Um, we can use some similarity mechanisms like vectoric search or something like that, much like RAG, or we can do some logic selection of the, of the, of the, of the example,

103. 25:21

     like by keyword or by category or whatever. Um, so this is the LLM Triangle. It, uh, composed, uh, by three, uh, key, uh, uh, principles, the model, the engineering techniques, and the contextual data, all guided by the SOP.

104. 25:44

     Again, data is the king. Without data, we don't have a kingdom.

105. 25:51

     Look at your data, try to think how you specialize it using the engineering techniques, the SOP,

106. 25:59

     to feed it to the model and to yield higher quality.

107. 26:04

     Thank you. Uh, if you have any question, please reach out to me over LinkedIn. Um, you can read the full white paper using this link, and thank you. Have a good one.