AI Engineer Summit 2025
Lessons from building GenAI based applications — Juan Peredo
Read the talk
Building a GenAI application beyond the first prompt
Generated code is only the beginning: reliable GenAI applications need deliberate hosting, evaluation, prompt management, agent orchestration, cost modeling, and traces.
From a talk by Juan Peredo
Before you start: Basic familiarity with web application development, API calls, and LLM prompts will help; retrieval and agent workflows are explained as they appear.
A prompt can start a website. Who finishes the application?
Give an LLM a prompt and get a website in a minute: the promise hides the difference between generating code and delivering an application. Juan Peredo opens with that gap. The model can produce a useful starting point, but the remaining work does not disappear.
Codeium and Cursor bring that assistance into an IDE; copying code into an LLM and asking a question is another workable interface. The appropriate model depends on the task. Routine website boilerplate asks less of a model than complex application logic or a recently released framework such as Svelte 5. For the latter, Peredo names DeepSeek V3 and Claude as stronger options. Faster coding helps, but integrating AI into the application’s user experience introduces its own engineering responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
AI adds a second set of engineering decisions
The conventional application still needs its frontend, backend, and infrastructure as code. Adding AI brings another set of choices: which model to use, whether to fine-tune it, how to write its prompts, and whether retrieval-augmented generation should supply information for its answers. Those choices also create a validation problem. A model’s willingness to answer does not establish that its answer is correct.
Model selection becomes an ongoing lifecycle, not a one-time dependency choice. Newly released models may perform well on benchmarks without working well inside the application already built around another model. Evaluate both the deployed model and potential replacements against the application’s actual needs. Hosting adds another dimension: CPU and memory planning may now include GPUs, with a choice between cloud and on-premises infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose where experimentation ends and operations begin
For exploration, Ollama lets you download models and interact with them locally without paying a hosted inference API for each request. The constraint moves to the machine: it must have enough capacity for the chosen model. The same category of tooling can support on-premises hosting. In the cloud, the question expands from which provider to use to whether suitable GPUs are available and what they cost. Peredo names Google Cloud, AWS, and Azure as familiar options.
Deployment tools can reduce the amount of infrastructure work around the model:
| Option | Mechanism | Useful when |
|---|---|---|
| Modal | Python decorators and CLI deployment; platform builds and pushes containers | Moving Python workloads into the cloud |
| SkyPilot | Configured cloud choices and workload placement | Seeking lower infrastructure costs across clouds |
| Hugging Face Transformers | Run models directly through a local library | Experimenting without an additional hosting tool |
Peredo describes Modal as a way to take Python code into cloud execution with little deployment ceremony. He describes SkyPilot as spreading clusters and containers across configured clouds to reduce cost. Transformers offers the more direct local path. These options simplify different parts of the job; none removes the need to choose a hosting arrangement that fits the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A working chatbot can still answer three times three with thirty-three
Peredo estimates that a basic chatbot built with existing frameworks can take one or two hours. Correctness is the harder problem. He reports asking an LLM what three times three was and receiving thirty-three. The interface worked; the answer failed. Validating and moderating what reaches the user therefore belongs to the application, even when a framework makes the chat interface easy to assemble.
Prompt engineering adds instructions and context to the user’s question. Those instructions need to be precise enough for the model to follow, but precision is not enforcement: the model may still disregard them.
A guardrail classifier adds a separate decision around the main model. In Peredo’s example, a secondary LLM classifies incoming questions as safe or problematic, including attempts to evade restrictions or requests involving criminal activity. The same pattern can classify outgoing answers. This creates another opportunity to catch an unwanted interaction, at the cost of another model call, additional latency, and another fallible judgment.
Retrieval-augmented generation, or RAG, supplies relevant company information when the model answers:
- Split documents into smaller chunks. Paragraphs are a useful illustration, not a required chunk format.
- Store the chunks in a vector database for retrieval.
- Use the incoming question to perform a similarity search.
- Put the retrieved information into the model’s context as material for answering the question.
The answer is now dependent on what the retrieval system contains and returns. If the database supplies outdated information, the model can produce an outdated answer while following the provided context. Data freshness is part of answer quality.
Fine-tuning changes the model through additional training. Peredo describes it as more expensive and slower than the preceding techniques, with no guarantee of improved responses. Poorly chosen training information can degrade behavior: the model may forget prior information or struggle with conflicts between old and new material. Prompting, guardrails, retrieval, and fine-tuning can be combined, but none makes chatbot correctness automatic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate candidates, providers, and recorded runs
Evaluation starts while the application is still an idea. Ollama supports local model trials; Hugging Face provides models and Spaces containing applications whose code and behavior can be inspected. These are ways to explore whether a model or interaction pattern fits before committing to an implementation.
OpenRouter extends that exploration to open and closed models through a common service. Peredo describes sending a prompt to Llama 3 and having the service choose the cheapest provider offering that model. For present-day implementations, that description needs one qualification: the documented default uses price-weighted load balancing with uptime considerations, rather than invariably selecting the cheapest endpoint; explicit sorting and provider controls are available.
Peredo reports depositing about $50 into OpenRouter six months earlier and having about $40 remaining despite daily exploratory use. He does not specify the workload. His production concern is more consequential than that personal spending example: an application should have safeguards around routing and visibility into which provider receives its requests. Convenient experimentation does not settle production provider policy.
LangSmith, from the creators of LangChain, extends evaluation into recorded application behavior. Recording LLM runs makes it possible to compare performance over time and see what the application and its models have actually done. Evaluation should continue through development and operation, rather than ending when a candidate model looks promising.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make prompts accessible to the people who understand the task
Hard-coding a prompt is convenient for the developer writing the first version. Externalizing it makes the behavior accessible to other people. Peredo uses LangChain Hub to illustrate a workflow in which an education expert, without programming experience, can edit a prompt, run it against an LLM, inspect the output, and commit the version they want the application to consume. The application must be wired to consume those externally managed prompts for that workflow to reach users.
External prompts also make model changes easier to handle. A prompt that works with one Llama release may produce different results with the next. Keeping prompts accessible for editing and testing shortens the adaptation loop. Peredo’s recommendation is to keep prompts out of embedded application code so that domain collaboration and model-specific iteration do not require editing the application’s logic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From answering a request to choosing functions
Agents extend the interaction beyond a chat response. Given the text of a book, an application might produce audio and translations into multiple languages. Peredo presents this as a promising direction still being developed: the model’s output can guide work that produces an artifact or changes something outside the conversation.
The simple agent diagram has a start, an assistant, tools, and an end. The assistant contains an LLM that analyzes the incoming request and chooses tool calls—function calls—until it has enough information or results to finish. In the book example, the assistant calls audio creation, then translation, then ends. The useful capability is choosing those actions without requiring the developer to hard-code every possible path through them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parallelize independent work to shorten the critical path
The freedom to choose actions introduces waiting time. In Peredo’s sequential example, four or five model calls taking three seconds each produce twelve to fifteen seconds of model latency. That wait conflicts with the responsiveness people expect from web applications; he warns that users may leave rather than wait.
LangChain, LlamaIndex, and LangFlow offer ways to organize branching and concurrent calls; the accompanying slide specifically names LangGraph. Return to the book request: node a can dispatch audio creation and translation independently, then node d can combine their results. The branches share the original book text. If the desired audio were instead a reading of the translation, that dependency would require a different ordering.
Python’s asyncio.gather expresses the independent branch operation directly:
python
import asyncio
from collections.abc import Awaitable, Callable
async def create_book_outputs(
book_text: str,
create_audio: Callable[[str], Awaitable[bytes]],
translate: Callable[[str], Awaitable[str]],
) -> tuple[bytes, str]:
audio, translation = await asyncio.gather(
create_audio(book_text),
translate(book_text),
)
return audio, translation
The split-and-rejoin diagram contains four work nodes: a, b, c, and d. With two seconds per node, running b and c concurrently reduces the illustrated duration from eight seconds to six. The critical path is a, then the slower branch, then d:
Concurrency helps where work is independent; it does not remove the time spent on the shared steps before and after the branches.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Price the complete customer interaction
Agent costs accumulate across the steps needed to serve one user. Peredo considers a hypothetical call center handling 3,000 customer calls per day, with 15 tool or function calls per customer call. He chooses OpenAI o1 for the illustration, quoting $15 per million input tokens and $60 per million output tokens, and assumes 1,500 input tokens and 3,000 output tokens per model request. The relevant unit is the whole customer interaction, not the price of a single token or request.
Peredo reports $3.24 per customer call, $9,270 per day, and almost $300,000 per month, but those figures do not reconcile. Treating each of the 15 steps as one identical billable model request, the stated rates and token counts instead yield $3.0375 per customer call, $9,112.50 per day, and $273,375 over 30 days.
The arithmetic correction preserves the lesson: apparently small token rates can become a substantial operating expense when multiplied across an agent’s repeated requests and the application’s traffic.
For provider endpoints billed by usage, additional users bring additional inference expense. Under a fixed workload per user, that expense scales linearly with traffic. Product pricing needs to cover the work the agent actually performs, including the repeated model interactions behind a single customer-facing action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Buy sufficient capability at a workable speed
Renting GPUs changes the billing model but does not eliminate infrastructure expense. Peredo gives an illustrative comparison of an A100 at potentially $4–5 per hour against a traditional EC2 instance costing fractions of a cent or a few cents per hour. These are broad examples without matching provider, region, instance configuration, or billing terms; they are not an equivalent-capacity comparison.
Peredo describes a spread of more than twentyfold between the endpoints of his model-price chart. That motivates a task-specific question: does this application require the expensive model’s capability, or could one or two cheaper models complete the same task? The evaluation needs to establish adequacy before the price comparison becomes actionable.
Output speed is another part of provider selection. For Llama 3.1 70B, Peredo reports Cerebras and Groq above 1,000 output tokens per second, Cerebras above 2,000, and unnamed providers at 31 and 29. Those are his chart comparisons, not established end-to-end application timings; the workload and measurement conditions are unspecified, and the Groq figure remains uncorroborated. The related Cerebras historical speed report provides context for the Cerebras claim.
Generation speed affects how long users wait, and model size also affects inference time. Peredo’s selection principle is to use a smaller model when it can do the job, associating that choice with faster responses, lower expense, and lower environmental impact. Capability that the task does not need can still impose a cost on every request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Revisit the call center with a cheaper model
The call-center example returns with Llama 3.3 70B replacing o1, provided the task does not require o1’s capabilities. Peredo reports the price falling from above $3 to $0.52 per customer call and from almost $300,000 to almost $50,000 per month. The slide identifies the replacement as Llama 3.3 70B on Groq and shows headline totals of $0.52 per call, $1,560 per day, and $46,800 per month. Those headline totals are consistent with 3,000 daily calls over 30 days, but the underlying replacement token rates are not supplied and the slide’s smaller calculation notes contain inconsistent totals. This is a price illustration, not a demonstration that the two models produce equivalent results.
The decision still turns on the use case. If the cheaper model meets the application’s requirements, its lower cost applies repeatedly across the workload. If it does not, the attractive price cannot make it an adequate replacement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trace the failure all the way back to the prompt
Multiple interacting model nodes make an agent difficult to debug. Unlike a fixed sequence of application operations, a probabilistic component introduces variation into the path and the values passed between steps. Peredo illustrates the surprise as a process that works repeatedly and then fails on a later attempt.
His concrete example is a database lookup for a user named John. The lookup had been working, then began failing. Traces revealed that the incoming name was lowercase john rather than John. Because the prompt had already been externalized, the team could add the instruction “ignore case.” Peredo reports that the lookup then worked again. The useful chain of evidence was the failing input, the behavior it triggered, and the prompt controlling that behavior.
The same lowercase input after a prompt repair
Constructed example: The display labels and isolated instruction field organize the anecdote for comparison. The complete original prompt, database schema, returned record, and trace payload were not supplied.
john
Operation: Add the instruction “ignore case” to the externalized lookup prompt.
Incoming name
john
john
User name being sought
John
John
Added prompt instruction
Not present
ignore case
Reported lookup outcome
Lookup fails
Lookup works again
In this multi-model agent, Peredo expects ordinary debugger sessions and print statements would have made diagnosis much harder. LangSmith is one way to obtain the needed visibility; custom instrumentation is another. What matters is having the execution evidence available when the intermittent failure occurs.
The closing trace example makes the required evidence concrete. The agent calls Llama Guard, then Llama 3.1, then Mistral. A useful trace preserves that order together with each call’s metadata, inputs, and outputs. When a step fails, it also retains the error and the information entering and leaving the model, so the developer can reconstruct what happened rather than infer it from the final response.
Observability is part of building the agent. The application needs a record of the path it actually took, especially when that path or its intermediate values can vary between requests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Introduction with a GPU inference example, Python decorators, container environments, and CLI execution.
Evaluate application outputs using datasets, compare experiments, and monitor production traces.
Model specifications and token pricing for the o1 model used in the call-center illustration.
Further reading
Historical report of 2,100 output tokens per second, including speculative decoding and expected speed variation.
Updates since the talk
Current controls for provider selection, price and performance sorting, and fallbacks.
Read the complete timestamped transcript
- 0:00
Hello, and welcome. Today we're gonna talk about the lessons I've learned from building GenAI application over the last year and a half.
- 0:10
My name is Juan Peredo, and I'll be your guide in this journey. I'm a founder, architect, consultant, developer, and everything in between. Uh, and I have over fifteen years experience, uh, in the IT industry.
- 0:25
And one of the first things, uh, everyone hears when they're talking about AI is about how much faster, uh, developers can code thanks to AI. You can hear this, and you've heard this before, I can guarantee, that nowadays you can just take a prompt, put it into an LLM, and you will-- an LLM will code your website
- 0:49
for you in about a minute or so. And we all know that that's not true, right? Um, it'll give you some code, and it'll get you going, but it won't build everything for you.
- 1:01
Um, but there's a lot of tools that have been created to help you with this, like some of the ones that I have listed here, like Codeium and Cursor, that interact with your IDE or an IDE by-- all by themselves, um, that help you, uh,
- 1:17
build your applications. You don't even have to use th-those tools, right? You can just take your code, paste it into an LLM, uh, and ask a question. All right?
- 1:27
And, and there's re-- some really good ones out there. Um, but it all depends on what type of application you're trying to build, right? If all you're doing is creating boilerplate, um, for yet another, uh, website, then any coding LLM will work just fine.
- 1:44
Um, if you're doing some more-- something more complex, or if it's something that's newer or just came out like, uh, I don't know, Svelte 5, for example, um, then you want a beefier LLM.
- 1:56
Uh, something like, uh, DeepSeek, uh, V3 or Claude. Right?
- 2:04
And, and these tools will definitely help you, um, build faster. Uh, however, they also, uh, add complexity to your application while you're trying to integrate AI to your application to actually interact with your user.
- 2:20
Like here in the left-hand side, we still see that in your typical, uh, application that we've been building till a year ago or two years ago, we were building frontends, backends, um, infrastructure as code, all the stuff that we normally build.
- 2:37
But the moment you add, uh, AI to your application, you al-- you have to start worrying about all the stuff that's in gray in the diagram on the right.
- 2:47
So you have to start off wondering about what model you're gonna use. Are you gonna fine-tune it? Are you gonna, um,
- 2:55
um, do some prompt engineering or are you try-- gonna try to use RAG to, uh, get the correct answer to your quest-- to, to your questions and to your user questions?
- 3:05
How are you gonna prevent it from hallucinating? Because we all know that, uh, LLMs are so bent on answering your questions that sometimes they'll just make up answers. Uh, and you need to...
- 3:18
a way to figure out, um, whether the answer that you're getting is right or not. And so how do you prevent that? That's something you didn't have to worry about a year a, and a half ago.
- 3:28
Um, also, you have to figure out, um, how often you're gonna have to, uh,
- 3:36
replace that model, because we all know that there's models that are coming out every other week, and they're all more powerful, beefier, but they all, they will not all work with your use case.
- 3:47
It's one thing that they're really great at certain benchmarks, but that doesn't mean they're gonna work, uh, with the rest of the application that you've built. So you have to constantly be ev-evaluating not only the model that you have already running but also the models that are coming out that may, uh, help you build a better application.
- 4:07
And you also have to start figuring out how you're gonna host these things now. Yeah, because that's beyond hosting just your application, right? All of a sudden, you don't have to worry just about CPU and memory.
- 4:18
All of a sudden, you have to start worry about GPU as well, right? Where are you gonna host this? Is it gonna be in the cloud? Is it gonna be, uh, on-prem?
- 4:27
All this information and all the stuff in gray is new to, uh, and is specific to AI-based applications.
- 4:39
Speaking of hosting, where to host this model is important, right? How are you gonna host it? Are you gonna host it in on-prem, or are you gonna host it on your machine while you're, um, doing some exploration?
- 4:52
You have to make these decisions, right? So if you're gonna host it locally, you can host it in something like Ollama. Ollama will allow you to, um, download a bunch of, uh, models into your machine where you can play with those models for free, uh, interact with them and see if they're gonna work for you.
- 5:09
And there's al- a host of other solutions as well. Ollama is the one I've been using. Uh, it's pretty good.
- 5:15
Just you just have to make sure you have a machine that's beefy enough, uh, for the model you're trying to run.
- 5:22
Or you can even use this Ollama and all these other tools, uh, to host things,
- 5:27
um, on-prem, right? But if you're gonna host it in the cloud, now you have to start thinking not only about what cloud, but also about the cost of your GPUs and is there enough GPUs to go around.
- 5:44
Uh, and in general, like e-everything else, you still al- have-- also have to worry about cost. And you have the, the typical players, right? You have your Google Cloud, your AWS, your Azure, et cetera, et cetera.
- 5:56
But there's also a, a new set of players that have come up, uh, that are
- 6:02
Meant to simplify the deployment of your application in the cloud. And that's where tools like Modal and SkyPilot, um, have come up.
- 6:16
Um, and if you're using Python, which a lot of us are using Python for, uh, interacting with your-- with our models, uh, Modal is a great choice. Modal allows you to take your Python code, add some, uh,
- 6:31
some, uh, uh, decorators to your code, and use their CLI to deploy, uh, to Modal. Uh, and Modal will take care of building your mo- your containers and pushing them to the cloud.
- 6:46
Um, it's really simple. I've used it to, to host a lot of my models.
- 6:52
Um, and there's also SkyPilot. SkyPilot is great to minimize your costs as they will spread the-- your clusters and your containers into multiple, uh, clouds in order to minimize your costs.
- 7:10
Right? And you can configure it in which, in which class you wa- cloud you want SkyPilot to, to use for hosting your application.
- 7:20
Um, and at the end of the day, right, if you just want to play with, with software in your machine, you can also use a, a simple library like the Hugging Face Transformers that you just can run, um, locally on your machine without any, any, any other tools.
- 7:34
Um, but at the end of the day, once again, how you host these models is very important.
- 7:44
Um, now le-let's think about, um, the fact that we're not in two thou- 2023 anymore, right? Building a chatbot powered by AI is very simple. You can build it in an hour or two, if that much.
- 8:00
All right? Um, there's a lot of frameworks there that will help you build this for you.
- 8:06
Um, however, even though building the chatbot is easy, making sure that that chatbot gives you the correct answers is really hard. Like, for example, this that I have on the screen, um, on the right is actually something that happened to me.
- 8:23
You-- I put in one LLM, uh, s- a few months ago now, what is three times three? And the answer that it gave me was thirty-three. Right? That's obviously not right.
- 8:34
Um, so while making the chatbot is easy, validating and moderating the chatbot content and making sure that it's giving you the correct answer to your user is really, really hard.
- 8:48
So what can we do to ensure that we are providing the right answer to your-- to our users, right? Well, there's a number of techniques. Right? You can do prompt engineering, which is basically you take the question of, of the user, you put additional, uh, details into the prompt that you pro-- pass into the, um, LLM.
- 9:10
However, you need to be very precise in those additional, uh, in those additional instructions
- 9:16
so that the LLM can follow them properly. And even then, the LLM may choose to ignore them. And at that point, what you do? Right? You get, you get the wrong answer.
- 9:27
You can also put guardrails. And guardrail is basically you take a secondary LLM and put it to run in front or before your main LLM. And this secondary LLM is basically a classifier that will classify, uh, the question that the ans- that the person is answering, um, as a
- 9:49
safe question or as a bad question. And when I say bad question, it could be, um, something that is trying to go a-a-around, uh,
- 9:59
uh, or asking for criminal activities, or it may be asking about, um, things that are, are not, not what you'd expect somebody to, to answer to, um, in general, right?
- 10:12
And at that point, the, the LLM is-- the, the secondary LLM is classifying the, your questions, and it also could be the, the answers o-of your own LLM as either good or bad.
- 10:23
However, you know, adding this secondary LLM adds la-latency because you're making a secondary call, right? And it also adds additional costs, and like anything else with LLMs, uh, it may or may not be right.
- 10:40
The next thing you could do is you could have Retrieval-augmented generation or RAG, which is basically you take a lot of information that you have in your company,
- 10:49
you break it into small chunks of information. So you can think of it of, uh, as, let's say, you take a document, and you break it into paragraphs, although it could be anything.
- 10:58
Just think of a chunk as a paragraph for now. You break it into chunks, and you store it in a vector database. And then when somebody asks a question, you take, um, you do a similari-similarity search on your, uh, vector database, pull the information, and provide that information to your, um, LLM, and tell the LLM, "Uh, here's
- 11:21
the information you can use to answer that question." And while that sounds really nice and good, it's really dependent on the type of information you provided in your vector database.
- 11:32
So if you put old information that's no longer relevant, well, guess what? What the LLM is gonna reply to you is with information that's no longer relevant. So data is extremely important in this case.
- 11:48
Um, the other thing you can do is you can do fine-tuning.
- 11:52
Right? And fine-tuning is basically you do additional training, uh, to an LLM.
- 11:58
This is more expensive than the other, the techniques that we're talking about, and it takes lo- a longer period of time.
- 12:05
And it doesn't really, uh, guarantee that you're gonna get r-good results. In fact, depending on the type of information you're putting, uh, i-into your fine-tuning, you could actually degrade the LLM quality responses,
- 12:20
um, because it may actually forget some of the information that it already had, or it may get confused between the new information you're passing and the information that it, that it already had.
- 12:31
So at the end of the day, there's no, no, no technique that will guarantee that the answer of your chatbot is gonna be good.
- 12:39
But, uh, using many of these techniques will help you get there, uh, or get you a long way.
- 12:51
Another thing that you should do is you should evaluate, uh, your models and your application
- 12:58
at each step of the development life cycle. For example, when you are starting to explore which LLMs to use in your, in your application or your idea of an application, you could use something like Ollama.
- 13:14
And we talked a little bit about Ollama before. It will let you run hundreds of LLMs locally on your machine.
- 13:22
Um, once you are interested in one or another, you can
- 13:28
start using it in your application. Or if you wanna play with thousands of, uh, open source models, you can go to Hugging Face,
- 13:38
and there you can build some Hugging Face space, spaces. Um, you can see the spaces that other people have built, a-and this is basically a lot of code that interacts with LLMs, um, and that you can freely use and review, um, to see whether you're interested to build something similar or, or use the, the LLMs that they're
- 13:57
using. Now, if you want to interact with other models, including, um, some of the closed models, you can go to OpenRouter. Now, OpenRouter will basically take your prompt, and it will run it against a number of models or a model of your choosing.
- 14:16
And one of the things that makes OpenRouter really cool is that it will, um, take your prompt, and let's say you want to run that prompt against Llama Three, it will look at the, uh, providers that are actually offering Llama Three as a, as an API, and it will run it until the one that's, uh, against the
- 14:38
one that's cheaper, which is really cool, and it's a great way to save money.
- 14:44
Um, I use it every day, and I put like fifty dollars about six months ago, and I still have about forty bucks left. And I use it, as I said, I use it every day, so it's a great way, um, to interact with, with models and, uh, see which one I want to use.
- 15:02
Uh, obviously, if you're gonna do so OpenRouter in production, uh, there are, there are a number of, um, safeguards that you, you may want to look into. You know, you don't want just, uh, OpenRouter just running into a, a random L- LLM provider without you knowing which one it is.
- 15:20
Um, but for testing, it's a great thing.
- 15:24
And finally, uh, the last tool I have here listed is LangSmith, which is created by the creators of, uh, LangChain, which is a very popular open source, um,
- 15:36
framework to interact with LLMs. But LangSmith, one of the great things about it is that it lets you constantly evaluate your models. So it will, uh, com--
- 15:49
It can help you record, uh, a number of runs that you have with your LLMs. You can compare over time, um, how your LLMs have, have been performing. Um, it's a really, really cool platform, um, just to keep track of what your application and your models are doing.
- 16:08
Um, so yeah, evaluation, you, you really should be doing this at each step of the way.
- 16:17
Another thing that it's very important is externalizing your prompts.
- 16:22
And I know that for all of us, it's really easy to just take a prompt, put it in the code, hard code it in there, and move on.
- 16:31
Um, but if you do that, you're shortchanging yourself. Externalizing your prompt, uh, in tools like, uh, repositories like Lang- LangChain Hub, which I have here, uh, a screenshot on the left, uh, will allow you to,
- 16:48
uh, share those prompts with experts. So if you are building an application that has to do with education, you could, you could reach out to, to the education expert that has never coded in their lives, and you can have them this-- log into, to the hub, play with, with the prompts, run the LLMs, see the output that
- 17:09
they're getting from the LLMs, um, and then when they're ready and they're happy with that they're getting the correct, uh, prompts, all they have to do is just, uh, commit those prompts, and that will flow directly into your application.
- 17:24
It's really cool. Another thing that will, that, uh, this will help you with is future-proofing. We all know that models are coming out every other week. There's a new model, and that's better than the previous one.
- 17:38
Um, and unfortunately, right, even let's say if you're using a Llama model and the next Llama model comes out, um, your prompt may not work prop-properly with the new version.
- 17:49
You may not get the, the same results. So by allowing you to easily access the prompts and modify them and test them against something new, then you c-- you are b-basically future-proofing and making your life easier in the future.
- 18:04
And that, in turn, leads to faster development.
- 18:08
So, bottom line, prompts should never be encoded in your code base.
- 18:17
I'll talk a little bit about agents. Agent, agents are,
- 18:22
and I expect them to be in two, uh-- this year, one of the best, uh, additions to the world of LLMs.
- 18:30
Because agents are basically allowing, um, the LLMs to break out of the chatbot and actually interact with the real world. So you can have an agent, for example, um, that, uh, allows you to...
- 18:47
For example, you have a book, you want to, uh, create the au-the audio for it, and you want to translate it into multiple languages. That's something that you can do, and you can have an output out of that, those things.
- 19:02
All right? So agents will allow us to interact with the real world. All these LLMs that have been confined to just chatting, all of a sudden will have a real impact in the world.
- 19:13
Um, and we're still not there. The-- We're still working on it, but it's a, a really ex-exciting, uh,
- 19:22
exciting, uh, set of applications that will be coming out soon.
- 19:28
And here I have a couple of, um, drawings, uh,
- 19:33
of what an agent looks like. And the one on the left,
- 19:38
that's, that has four nodes, so it has start nodes, assistant tools, and it's really, really powerful.
- 19:45
Right? Because basically what you're doing is you're--
- 19:49
When somebody gives you a question, you pass it to the assistant, and the assistant w- basically has an LLM embedded. And that LLM will analyze the question
- 20:00
and then will start calling tools. And tools are-- it's a fancy name for functions. It would start calling functions, um, until it gets to the answer that it, that it needs, uh, to answer the question of the end user.
- 20:15
So, for example, it could be that the end user sends, uh, the text of a book to the assistant, and it tells it to translate it and, uh, create the audio, and then the assistant will call the tool to create the audio.
- 20:29
Then it will call the tool, uh, to the translation, and then it will end.
- 20:34
Right? And the assistant is making those decisions all by itself, which is a, a really, really powerful, um, concept.
- 20:45
Um, and while it's really cool that those agents can make those decisions without you actually having to hard code all those paths,
- 20:54
um, they do have its disadvantages, right? And one of the big ones is that LLMs have a lot of latency. You know, calling one of th-those LLMs takes several seconds,
- 21:07
right? So all of a sudden, if you have an agent that is integrating with four or five LLMs, and each one of them takes, let's say,
- 21:20
three seconds to run, then all of a sudden you have,
- 21:24
uh, a process that takes twelve, fifteen seconds, right? And we know that our users nowadays are used to running things on the web and taking those things taking only milliseconds, right?
- 21:39
And if they have to wait for a second or two, people are leaving our sites and going to our competition, which is not the best, uh, result.
- 21:48
Um, so we should take advantage of, o-of, uh, these platforms that have been built like LangChain, LlamaIndex, LangFlow. There, there's many of them, right?
- 22:00
That allow us to do things like concurrent calls and branching, uh,
- 22:05
in these agents, right? So, like, for example, I have here the, on, on the left a diagram of an agent that's actually calling, uh, a couple of things in, in parallel, right?
- 22:17
So it could be, once again, it could-- this could be, um, the user passing, uh, a book to node A. Node A calling, uh, the audio translate-- the audio creation and the translation in parallel, and then sending the, uh, answer back to the user.
- 22:37
Now, if each one of these things took, uh, two seconds, instead of these things taking eight seconds, all of a sudden it takes six,
- 22:45
right? So that's much better, right? So there's a lot of things and, and, and a lot of things that, that need to come to mind when you're building these agents, so that you can make them as efficient as possible, so that we can still get to the end to fulfill the, the expectations of our users
- 23:07
while taking advantage of the LLM's powers. Um, and while I do believe that agents are the future for LLMs, we really have to be aware of the costs of running these agents,
- 23:25
um, because they can-- the cost can add up really quickly. Let's take, for example, the, the one I have here where once again, it's a simple, um,
- 23:36
agent. Um, and we're, we're gonna assume that this is a call center, right? This call center takes three thousand calls a day, and each one of those calls, uh, requires fifteen, uh, function calls or tool calls.
- 23:53
And we're gonna assume that we're using OpenAI o1. It's a great model, right? So we're gonna take advantage of it because it's the-- one of the big-biggest and greatest.
- 24:04
Um, and the prices for, for OpenAI o1 is fifty million-- fifteen dollars per million tokens or sixty dollars per million tokens out.
- 24:15
And if we assume that we're inputting fifteen hundred input tokens and three thousand output tokens, meaning, uh, we're sending a bunch of information into the LLM, the LLM is impor-- is, uh, is, uh, outputting a bunch of information out.
- 24:32
Um, let's figure out how much that will cost us, right? Uh, 'cause it sounds like very little, fifteen dollars per one million token, right? It sounds like it shouldn't be much.
- 24:45
But once you do the math, and the math is at the bottom if you wanted to take a look at it. We won't discuss it here. Uh, but it comes out to three dollars and twenty-four cents, uh, per call, which turns into nine thousand two hundred and seventy per day, and which, uh, tu-- i-in turn is almost
- 25:01
three hundred thousand dollars per month. Um, so if you were expecting to get a very small price tag, um, well, you're gonna be-- have a little bit of a surprise. [laughs]
- 25:16
Uh, at the end of the day, right, especially if you're using, um,
- 25:23
provider endpoints for your APIs to interact with the LLMs, you have to keep in mind that the cost of running this thing
- 25:32
is linear, right? So each additional user will incur you an additional cost. Now, obviously, you have to make sure that y-y-you're, you're, you're pricing your, um, products accordingly. But if you were expecting to get very low prices, uh, and very low cost, uh, you may want to, uh, do the math.
- 25:57
Um, but even if you are renting s-- uh, the GPUs, right, so you don't have to make the API calls. GPUs are at a premium nowadays. So, um,
- 26:11
if we compare the price of a GPU versus the price of a CPU on the cloud, we're talking cents versus dollars. All right? So having an A100, uh, GPU for an hour will cost you like potentially four or five dollars an hour versus, you know, a, a traditional EC2 instance that could cost you,
- 26:36
uh, a fraction of a cent or a couple of cents an hour. So keep that in mind.
- 26:45
A-and the choice of the model a-and the provider that you're gonna use in your im-- in your application is really, really important. And we can see in the price chart on the left that there's a significant va-value difference between the, uh, left, so the first LLM in the list and the last one, right?
- 27:06
It's, it's over twenty times difference in terms of price, right? Um,
- 27:14
so you have to evaluate, do you really need the power of a very pricey LLM, or can you do the same thing with a cheaper
- 27:26
LLM or two that may not be quite as powerful, but they can do the task, right? It might be, from a price perspective, it might be much, much, much more ef-efficient, right?
- 27:40
The other thing that's important, like we said, is, uh, output speed, right? How fast these, these models, um, can, uh, respond to questions.
- 27:52
And that's what we see in the chart on the right, the output speed. And we can see that Cerebras a-and Groq can run Llama three point one seventy B at over a thousand, uh, tokens per second.
- 28:09
In fact, Cerebras is over two thousand tokens per second. While all the way on the right, the, the,
- 28:17
the providers are running thirty-one and twenty-nine tokens per second,
- 28:22
right? All that translates to latency for your users. And usually, the size of the, of the model that you're picking also has a big, a big impact, right? So bigger models will take run-- longer to run.
- 28:35
Smaller models will go faster. Which one you're using and which one you need to use will have a big impact on, on your application.
- 28:46
So at the end of the day, right,
- 28:48
small is beautiful and better for the environment and better for your wallet.
- 28:57
So let's go back to our example, right?
- 29:01
Now, what happens if we replace OpenAI o1 with Llama three point three seventy B just from a price perspective, right? You may need to use OpenAI for your use case, but what if you didn't, right?
- 29:12
You can just use, uh, s-- Llama three point three. Um, well, we go from three... over three dollars per call to fifty-two cents per call. And we go
- 29:25
all-- from almost three hundred thousand dollars per month to a l-- almost fifty thousand dollars per month.
- 29:35
So once again, pick the right model for the use case.
- 29:42
At the end of the day, your wallet and the environment will thank you.
- 29:51
And the last thing I wanna touch on is observability for your agents. I can't tell you how hard it is, uh, to try to figure out what an agent is doing once you get multiple nodes that are interacting with each other.
- 30:07
Uh, this is a lot harder in my experience than trying to figure out what a regular application is doing because a regular application is deterministic. It will do A, B, C, D.
- 30:19
Well, um, a probabilistic model adds lots of unknowns to your application. So all of a sudden, things that were running a hundred times with no issue would start failing on a hundred, uh, a hundred and first item, right?
- 30:41
And I'll give you an example, right? I was running an LLM, and all it had to do was get the information about certain, uh, a user from the database, right?
- 30:51
So let's say, for example, get information for user John, and this would run
- 30:59
for the longest time. And all of a sudden it started failing.
- 31:03
And we realized that eventually, because, because we had the traces, um, that the user was entering lower cap Jo- John instead of just capital J and the rest of the word.
- 31:16
Okay? Uh, and all it took was going back to the prompt, which we, we had externalized lu- luckily, and changing the prompt for, uh... to add the sentence, uh, ignore case, and everything started working again.
- 31:30
But because this one, uh, was an agentic application and it was calling multiple, uh, models,
- 31:38
it would have taken us a very long time to figure out what was going on. I mean, we could put, uh, we could, we could have put the debugger and print statements, all that, and it would still have been very, very hard.
- 31:51
So ha- have observability for agent. There's a lot of tools out there like LangSmith that will make this very simple. Uh,
- 32:02
and if you don't wanna use one of the tools that's available there, um, you can build one or you can build all this. But build it because you are going to need it.
- 32:10
Um, and in here, um, we have our agent traces. On the left,
- 32:16
um, of the screen, we have an agent that's first calling Llama Guard, then it's calling Llama 3.1, then it's calling, calling Mistral. And that ha--
- 32:26
that, that gives us an order of everything that, that, that had run in that, in that particular call to the agent. It has metadata, it has the inputs and the outputs.
- 32:37
All that is crucial, right? And when it, when it throws an error, it keeps track of the errors, right? All the information that was in and out of that LLM so you can see exactly what happened, right?
- 32:49
And you can see the error, uh, the trace like we see, uh, on the trace on the picture on the right.
- 32:56
Yeah. So, uh, like I said, building observability for your agents is really important.
- 33:06
And with that, any questions?