AI Engineer World's Fair 2025
POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments
Read the talk
From GenAI Proof of Concept to a Product People Can Use
Video search, sports highlights, cost dashboards, and hospital workflows show how specifications, context, evaluations, and interface design turn model capabilities into useful products.
From a talk by Randall Hunt
Before you start: Familiarity with LLM prompts, embeddings, and basic application architecture will help with the retrieval and deployment examples.
Start with the customer’s operating problem
What does generative AI actually contribute when a customer needs an application built or an Oracle database moved to Postgres? Caylent works across those kinds of requests, serving startups through large enterprises. That breadth gives Randall Hunt a practical starting point: generative AI is one component of a solution, not a universal answer to whatever problem arrives next.
Building operations provide one example. BrainBox AI manages HVAC systems, and Caylent built an agent to support its work on building management and decarbonization. Hunt describes a North American footprint of tens of thousands of buildings; that is his account of the deployment scale. He also recalls recognition from TIME: ARIA appeared in its 200 Best Inventions of 2024. Water management and conservation provide another application. These are operational problems with a purpose beyond adding a conversational interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make stock footage searchable through language and images
Nature Footage has a large stock-video collection: lions, tigers, bears, crocodiles, and other wildlife. Making it searchable requires more than putting one vector beside each file. The system also needs descriptions that capture what happens in a clip and when it happens. Nova Pro generates video descriptions, timestamps, and features, which the application stores in Elasticsearch for retrieval.
The visual retrieval path samples frames from a video, embeds those frames, and pools their embeddings into a representation of the clip. A text query can then retrieve visual content through a shared multimodal embedding space. Descriptions and vectors serve complementary purposes: one makes the video’s interpreted content searchable, while the other supports similarity across text and images. Hunt calls the embedding model “Titan V2 multimodal”; AWS documents its image/text model as Titan Multimodal Embeddings G1, distinct from Titan Text Embeddings V2. That naming distinction does not establish which model was deployed in this example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find sports events, then simplify what the model must see
A second video architecture processes both live sports and archival footage. Its pipeline combines conventional signal processing with model inference:
- Extract the audio and generate a transcript.
- Use FFmpeg to inspect audio amplitude and look for audience cheering as a simple highlight heuristic.
- Generate embeddings from the text and video, identify behaviors with confidence estimates, and store the results.
- Send event notifications through Amazon End User Messaging or SNS—for example, when the system identifies a three-pointer.
Cheering is a useful way to find candidate moments before asking a model to interpret them. It does not require the model to discover every interesting interval from raw footage alone.
The next improvement changes the model’s input. With a static camera angle, annotate the court’s three-point boundary with a prominent blue line. The question can then become: did the player cross the blue line? This makes the relevant boundary explicit instead of requiring the model to infer it from the unmodified scene. Hunt reports better results with this small augmentation, without giving a numerical benchmark. SAM 2 can assist with annotations; he presents it as an option, not as a verified component of this particular deployment. The architecture also names Postgres with pgvector and OpenSearch as vector-store choices.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate productivity, process savings, and monetization
Hunt’s delivery perspective spans physics work at NASA, database technology at 10gen before it became MongoDB, leadership of SpaceX’s CI/CD team, and customer engineering at AWS. He also recalls making a Transformer explainer in July 2017. At Caylent, that background feeds a customer-building practice covering chatbots, copilots, and agents. The useful distinction is what each application is supposed to accomplish.
Self-service productivity often starts with software an organization can buy. Some institutions still need a fine-tune or an application around it, but they also need to administer third-party tools, track API usage, and control sensitive data. Hunt describes on-premises networks and VPNs as places to inspect traffic for personally identifiable information and protected health information. He recommends SurePath, which Caylent uses, and mentions integration with Zscaler.
Business-process automation has a different target: recover time or money across an entire workflow. For a logistics customer handling receipts and bills of lading, Caylent puts a custom classifier before the generative models. Hunt reports faster and better results than that customer’s human annotators, but supplies no measured comparison. The architectural point is the sequence: classify the document before sending it into generative processing.
Monetization adds a paid capability to an existing product, often as a new SKU. An existing SaaS platform already has users and a workflow in which AI might create value. A generic chatbot, however, gives those users little reason to prefer one vendor’s implementation over another. The feature needs differentiation beyond the presence of a model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define the contract before choosing the model
The system’s inputs and expected outputs are its fundamental contract. Start there, then build evaluations that distinguish repeatable behavior from a successful response to one unusually favorable prompt. A demonstration shows that something can work; an evaluation set begins to establish whether it keeps working as the inputs change.
The hierarchy then moves through system architecture to particular models and tools. Those implementation choices should be expected to evolve. The specification can also expand as new output modalities become practical, but the business questions remain: why build this system, what result should it produce, and what return justifies the investment? That keeps model selection subordinate to the product’s purpose.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose compute and storage with their costs attached
In the AWS stack Hunt presents, Bedrock and SageMaker form the service layer. He describes SageMaker as carrying a compute premium; running on EKS or EC2 is another option. Below that sit accelerator choices. Hunt cites approximately 60% better price-performance for Trainium and Inferentia than NVIDIA GPUs, without specifying workloads, chip generations, or a GPU baseline. He pairs that claim with a warning about less HBM capacity than an H200, likewise without identifying the AWS accelerator generation. Treat these as prompts for a workload-specific comparison, not portable benchmark results.
Hunt also points to newly announced P4 and P5 price cuts, recalling a maximum of 40%. The June 2025 AWS announcement actually advertised up to 45%, with that maximum applying to P5 three-year EC2 Instance Savings Plans. These are historical purchasing terms, not current prices. Trainium and Inferentia also introduce a programming requirement: the Neuron SDK. Hunt compares the Neuron Kernel Interface with XLA and TPU programming. Above the compute layer, model choices include Claude, Nova, Llama, DeepSeek, and other deployable open-source models.
Vector storage carries its own economic tradeoffs. Hunt prefers Postgres, while also pointing to MemoryDB for Redis-compatible persistence and vector search on AWS.
| Store | Useful property | Cost consideration |
|---|---|---|
| MemoryDB / Redis vector search | Fast, RAM-resident search | Memory expense |
| Postgres with pgvector | Database and vector retrieval together | Index and memory planning |
| OpenSearch | Search and vector retrieval together | Index and storage planning |
The central issue is where the index lives and how much memory it needs. Hunt warns that IVFFlat-style index planning can consume substantial RAM, then points to disk-backed storage and HNSW options in Postgres and OpenSearch. HNSW is not a promise that memory requirements disappear.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Context and access patterns make retrieval useful
Prompt versioning and management are common infrastructure. Context management is a stronger source of differentiation: an application may know which page a user is viewing, what they have browsed, and other information relevant to the current task. Supplying that context lets the model make an inference grounded in the user’s situation rather than answering the same generic question for everyone.
Evals and embeddings still do not define a complete product. The design must reflect how people actually access information. A similarity query alone does not provide faceted navigation or structured filters. Postgres and OpenSearch are useful here because retrieval can combine semantic matching with the other constraints users need to express. Start with the access pattern, then choose the retrieval machinery that supports it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce waiting and avoid unnecessary model work
Latency affects adoption alongside price. Caching can reduce actual waiting, while interface feedback can make unavoidable waiting easier to tolerate. Hunt argues that a slower, more expensive system is difficult to justify, whereas a slower but cheaper system may still succeed if its interface handles the delay well. The user’s experience includes the wait, not just the final answer.
Some delays are self-inflicted. A tool whose only job is to return the current date asks the model to request information the application already knows how to obtain. Supply it directly in the prompt instead. For example, application-side Python can construct the context without a model tool call:
python
from datetime import datetime, timezone
instructions = "Answer the user's question using the supplied context."
current_date = datetime.now(timezone.utc).date().isoformat()
prompt = f"{instructions}\n\nCurrent date (UTC): {current_date}"
Placement matters too. Frequently changing information near the beginning of a prompt can reduce prefix-cache reuse. Keeping stable instructions first and placing dynamic context afterward can preserve more reusable prompt content.
As models improved, prompt engineering proved more effective for Caylent than Hunt had expected, changing his earlier preference for fine-tuning. Moving identical prompts from Claude 3.5 to Claude 3.7 caused regressions in some customer cases and evaluations. Hunt reports zero regressions in Caylent’s observed move from Claude 3.7 to Claude 4. He also describes faster, better, cheaper inference in virtually every use case they encountered, without supplying benchmark conditions. That is an account of their upgrade experience, not a guarantee for another application or a future release.
Less prompt rework on future upgrades is a hope, not an established property of model releases. Each application still has to justify its inference economics. A capable Opus model may be inappropriate if its cost makes the feature unsustainable at the company’s expected usage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn the first manual test into an evaluation set
Evaluation does not have to begin with an elaborate scoring framework. Using a slide he attributes to Anthropic, Hunt describes an incremental procedure:
- Preserve the first manual test—the initial qualitative check—as an evaluation case.
- Change the input data to create additional cases.
- Define what success means. A Boolean result may be easier to judge than an artificial numerical score.
- Run the cases, inspect the outputs, adjust the prompt, and repeat.
The useful transition is from an impression of one answer to a repeatable test of behavior. A small initial set gives the team something concrete to run as the system changes.
Continue iterating across output evaluation, prompt management, and the interface. The slide extends the process through A/B evaluation and feedback loops. Production readiness emerges from those repeated checks, while UX orchestration determines whether the system’s behavior feels useful to the person waiting for it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit the response to the user’s work
CloudZero began with a chatbot for asking questions about AWS infrastructure costs. The application then moved toward generative UI: create a React component just in time and insert it into the response rendering so the answer can include an appropriate chart or other presentation. Generated components can be cached, and later prompts can describe available components for possible reuse. Hunt presents this as a way for the interface to evolve and personalize, made practical by sufficiently fast inference.
For users in remote areas, the limiting factor was connectivity rather than the sophistication of the answer. They received useful summaries of manuals, then faced approximately 200 MB PDF downloads to see the supporting material. Caylent’s adaptation was to render the relevant PDF page on the server and send that page image with the text summary. The user could read the necessary instructions and inspect the relevant source page without transferring the full manual. This solves a different interface problem from CloudZero’s charts or Nature Footage’s similar-video search, even though all three use AI within the product.
A hospital deployment exposed another mismatch. Caylent initially built a voice bot for nurses, but the noisy environment interfered with transcription and picked up other people speaking. The nurses in that deployment preferred a conventional chat interface. Knowing the user means understanding the conditions in which they work, not merely choosing the newest input modality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend inference only where it contributes
Ordinary computation should handle arithmetic. Asking an LLM to calculate spends inference on work a computer can perform directly. Prompt optimization also needs to account for output tokens: the volume of generated text affects cost and performance, so producing the needed answer efficiently is part of the engineering task.
Other cost controls include prompt caching, appropriate tool use, and batch processing. Hunt cites a 50% discount for Bedrock batch inference; the historical announcement limits that discount to selected supported models and regions, relative to on-demand inference. It applies to asynchronous work, rather than establishing a universal discount for every model or interactive request.
Finally, seek minimum viable context: the smallest context that still supports a correct inference. That requires knowing what the user is doing, injecting the information that matters, and removing material that does not. Context optimization is an ongoing process of deciding what the model needs to reason over—not simply making every prompt shorter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Meta's image and video segmentation code, model checkpoints, and example notebooks.
The open-source extension for vector similarity search inside Postgres.
Historical P4 and P5 price reductions, broken down by instance type and purchasing plan.
Further reading
AWS documentation for embedding images and text into a shared space for search and recommendations.
The historical announcement of discounted asynchronous inference for selected Bedrock models.
Updates since the talk
- POC to PROD, Part 1Article
Hunt and Ojeda expand the deployment lessons into specifications, evaluations, interface design, and inference economics.
- POC to PROD, Part 2Article
A companion discussion of production architecture and multimodal video retrieval patterns.
Read the complete timestamped transcript
- 0:00
[upbeat music] Everybody excited.
- 0:15
So, uh, what does Caylent do? We build stuff for people. So people come to us with ideas and they're like, "Ah, I want to make an app," or like, "Oh, I want to move off of Oracle onto Postgres."
- 0:23
So you know, and we just do that stuff. We are builders. We, uh, created a company by hiring a bunch of passionate autodidacts with a little bit of product ADHD, and we jump around to all these different things and build cool things for our customers.
- 0:36
And we have hundreds of customers at any given time, everyone from, like, the Fortune 500 to startups, and it's a very fun gig. It's really cool. You get exposed to a lot of technology.
- 0:46
And what we've learned is that, uh, generative AI is not the, the magical pill that solves everything that a lot of people seem to think it is. Uh, and then what your CTO read in The Wall Street Journal is not necessarily the latest and greatest thing.
- 1:01
And we'll share some concrete components of that. Uh, but I'll just point out a couple of different customers here. One of the ones is BrainBox AI. So they are a, uh, building operating system.
- 1:14
They help decarbonize the built environment, so they manage, uh, tens of thousands of buildings across the United States and Canada or North America, and they manage the HVAC systems.
- 1:25
And w- we built an agent for them for helping, uh, with that, uh, decarbonization of the built environment and managing those things. And that was, uh, m- I think in The Times 100 Best Inventions of the Year or something because it helps drastically reduce greenhouse emissions.
- 1:41
Uh, and then Simmons is, uh, water management conservation, which we also implemented with AI. Uh, and with that, you know, there's a couple other customers here, Pipes AI, Virtual Moving Technologies, D5 Inventory.
- 1:52
Uh, but I thought it'd be cool to just show a demo. And one of the things that I'm most interested in right now is multimodal search and, uh, semantic understanding of videos.
- 2:01
So this is one of our customers, Nature Footage. They have, uh, a ton of stock footage of, you know, lions and tigers and bears, oh my, and crocodiles, I suppose.
- 2:11
And we needed to index all of that and make it searchable over, uh, not just a vector index, but also, like, a caption. So we leveraged the Nova Pro models to generate understandings and timestamps and features of these videos, store all of those in Elasticsearch, and then we are able to search on them.
- 2:30
And one of the most important things there is that we were able to build, uh, a pooling embedding. So by taking frame samples and pooling the embeddings, uh, of those frames, we can do a multimodal embedding and search with text for the images, and that's provided through the Titan V2 multimodal embeddings.
- 2:46
So, uh, I thought we'd take a look at a different architecture.
- 2:51
I hope no one here is from Michigan because that's a terrible team. I hate them. Anyway, anyone remember March Madness? So this is another customer of ours that, uh, I'm not going to reveal their name, but essentially we have a ton of sports footage that we're processing both in real time and in batch, archival and in real
- 3:07
time. And what we'll do is we'll split that data into the audio. We'll generate the transcription. Fun fact, if you're looking for highlights, the easiest thing to do is just FFmpeg, get an amplitude spectrograph of the audio, and look for the audience cheering, and lo and behold, you have your highlight reel.
- 3:21
Um, very simple hack right there. And we'll take that, and we'll generate embeddings from both the text and from the video itself. And we'll be able to, uh, identify certain behaviors with a c- certain vector and a certain confidence, and we'll store those then into a database.
- 3:37
Oh, I think I paused the video by accident. My apologies. No, I didn't. And then we'll use something like Amazon End User Messaging or SNS or whatever, and we'll send a push notification to our end users and say, "Look, we found, uh, a, a three-pointer," or, uh, "We found this other thing."
- 3:53
And what we found is, um, you don't even have to take the m- the raw video. A, a tiny little bit of annotation can do wonders, um, for the video understanding models a- at, the, as they exist right now.
- 4:05
The SOTA models still just with a little tiny bit of, uh, augmentation on the video will outperform, um, what you can get with an unmodified video. And what I mean by that is if you have static camera angles and you annotate on the court where the three-pointer line is with a big blue line, and then you just
- 4:23
ask the model questions like, "Did the player cross the big blue line?" Lo and behold, you get way better results. And it takes, you know, seconds, and you can even have something like SAM 2, which is another model from Meta, go and do some of those annotations for you.
- 4:35
So that's an architecture. You'll notice that I've put up a couple of different databases there. We had, uh, Postgres, pgvector, uh, which is my favorite right now. We had OpenSearch.
- 4:46
Uh, that's another implementation of vector search there. Um, but anyway, why should you listen to me? Hi, I'm Randall. Um, I got started out hacking and building stuff and, uh, playing video games and hacking into video games.
- 4:59
It turns out that's super illegal. Did not know that. Um, and then I went on to do some physics stuff at NASA. Uh, I joined a small company called 10gen, which became MongoDB.
- 5:09
Uh, they IPO'd. Um, I was an idiot and sold all my stock before the IPO. Uh, and then I worked at SpaceX, where I led the CI/CD team. Fun fact, we never blew up a rocket while I was in charge of that team.
- 5:20
Before and after my tenure, we blew up rockets. Um, I, I don't know what else I can say there. Uh, and then I spent a long time at AWS, and I had a great time building a ton of technology for a lot of customers.
- 5:30
I even made a video about the Transformer paper in July of 2017, not realizing what it was gonna lead to and the fact that we're all even here today is, is still attention is all you need.
- 5:43
Uh, you can follow me on Twitter [REDACTED:username]. Uh, it's still called Twitter. It will never be called X in my mind. And, uh, this is Caylent. You know, we've won partner of the year for AWS for a long time.
- 5:52
We build stuff, like I said. I, I, I like to say our motto is, "We build cool stuff." Um, marketing doesn't like it when I say that, uh, 'cause I don't always say the word stuff.
- 6:00
Sometimes I'll sub in a different word.
- 6:03
And what we build, you know, everything from chatbots to copilots to AI agents, and I'm gonna share all the lessons that we've learned from building all of these things.
- 6:11
You know, this sort of stuff on the top here, these self-service productivity tools, um, these are things that you can typically buy, uh, but certain institutions may need a fine tune.
- 6:22
They may need a- a particular application on top of that self-service productivity tool, and we will often build things for them. Uh, one of the issues that we see organizations facing is how do they administer and track the usage of these third-party tools and APIs?
- 6:37
Uh, and some people have an on-prem network and a VPN where they can just measure all the traffic, they can intercept things, they can look for PII or PHI, and they can do all the fun stuff that we're supposed to do with network interception.
- 6:46
There's a great tool called SurePath. Uh, we use it at Caylent. I recommend them. Uh, it does all of that for you, and it can integrate with Zscaler or whatever else you might need.
- 6:55
Um, in terms of automating business functions, you know, this is typically trying to get a- a percentage of time or dollars back, uh, end to end in a particular business process.
- 7:07
Uh, we work with a large logistics management customer that does a tremendous amount of processing of, uh, of receipts and bills of laden and things like that. And this is a typical intelligent document processing use case.
- 7:19
Leveraging generative AI and a custom classifier before we send it into the generative AI models, uh, we can get far faster, better results than even their human annotators can.
- 7:30
Um, and then there's monetization, which is adding a new SKU to an existing product. It's an existing SaaS platform, it's an existing, uh, utility, and the customer is like, "Oh, I wanna add a new SKU so I can charge my users for fancy AI because The Wall Street Journal told me to."
- 7:44
And that is a very fun area to work in, but if you just build a chatbot, you know, sayonara. Like, good luck, I'll s- you know, you're the Polaroid.
- 7:55
Um, do people still use Polaroid? Are they doing okay? I don't know. Anyway, I used to say Kodak. Um, this is how we build these things, and these are the lessons that we've learned.
- 8:04
Um, I stole this slide. This is not my slide. I cannot remember where it is from. It's from Twitter somewhere. It might've been Jason Wu, it might've been from DSPy, but this is a great slide that I think very strategically identifies what the, uh, specifications are to build a moat in your business.
- 8:22
And the inputs to your system and what your system is going to do with them, that is the most fundamental part, your inputs and your outputs. Um, does everyone remember Steve Ballmer, uh, the former CEO of Microsoft, and how he, uh, famously went on stage, uh, on a tremendous amount of cocaine and just started screaming, um, "Developers.
- 8:41
Developers. Developers. Developers"? If I were to channel my inner Ballmer, what I would say is, "Evals. Evals. Evals. Evals." So when we do this evals layer, this is where we prove that the system is robust and not just a- a vibe check, and we're getting a one-off on a particularly unique, uh, prompt.
- 9:01
Then we have the system architecture, and then we have the different LLMs and tools and things we may use. And these are all incidental to your AI system, and you should expect them to evolve and change.
- 9:10
What will not evolve and change is your fundamental definition and specification of what are your inputs and what are your outputs. Uh, and as, you know, the models get better and they improve and you can get other, like, modalities of output, that may evolve.
- 9:24
But you're always gonna figure out, why am I doing this? What is my ROI? What do I expect?
- 9:30
This is how we build these things in AWS. On the bottom layer we have two services. We have Bedrock and we have SageMaker. Uh, these are, uh, useful services.
- 9:39
SageMaker comes at a particular compute premium. You can also just run on EKS or EC2 if you want. Um, there's two different pieces of custom silicon that exist within AWS.
- 9:49
One is Trainium, one is Inferentia. Uh, these come at about a- a sixty percent price performance improvement over using NVIDIA GPUs. Now, the downside is the amount of HBM RAM is not as big as, like, an H200.
- 10:01
I don't know if anyone saw today, but it was great news. Amazon announced that they were reducing the prices of the P4 and P5 instances by up to forty percent, so we all get more GPUs cheaper.
- 10:10
Very happy about that. Um, the interesting thing with Trainium and Inferentia is that you must, uh, use something called the Neuron SDK to write these. So if anyone has ever written XLA for, like, TensorFlow and the good old, um, what were they called?
- 10:24
The TPUs, and now the new TPU7 and all that great stuff, uh, the- the neuron kernel interface for Trainium and Inferentia is very similar. One level up from that, we get to pick our various models.
- 10:33
So we have everything from, uh, Claude and Nova to Llama and DeepSeek, uh, and then open source models that we can deploy. I don't know if Mistral is ever gonna release another open source model, but who knows?
- 10:44
Uh, and then we have our embeddings and our vector stores. So like I said, uh, I do prefer Postgres right now. If you need, um, persistence in Redis, uh, there's a great thing called MemoryDB on AWS that also supports vector search.
- 10:57
Um, the good news about the Redis vector search is that it is extremely fast. The bad news is that it is extremely expensive because it has to sit in RAM.
- 11:05
Um, so if you think about how you're going to construct your indexes and, like, do IVV flat or something, uh, be prepared to blow up your RAM in order to store all of that stuff.
- 11:13
Now, um, within Postgres and OpenSearch, you can go to disk and you can use things like HNSW indexes so that you can have, uh, a better allocation and search mechanism.
- 11:22
Then we have the prompt versioning and prompt management. Uh, all of these things are incidental and- and kind of, uh, uni- you know, not unique anymore. But this one, context management, is incredibly important.
- 11:35
And if you are looking to differentiate your application from someone else's application, context is key. So if your competitor doesn't have the context of the user and additional information, uh, but you're able to inject, oh, the, the user is on this page, they have a history of this browsing, you know, these are the cookies that I saw,
- 11:54
this is a... Y- you know, then you can go and make a much more strategic inference on behalf of that end user. So here are the lessons that we learned, and I'll- I'll jump into these, but I'm also gonna run out of time, so I'll- I'll speed through a little bit of it and I'll make this deck
- 12:07
available for folks. But-- Uh, it turns out evals and embeddings are not all you need. Uh,
- 12:14
you know, the, understanding the access patterns and understanding the way that people will use the product, uh, will lead to a much better result than just s- throwing out evals, and throwing out embeddings, and wishing the best of luck.
- 12:25
Embeddings alone do not a great query system make. How do you do faceted search and filters on top of embeddings alone? That is why we love things like OpenSearch and Postgres.
- 12:35
Um, speed matters. So if your inference is slow, sayonara. Uh, UX is a means of mitigating the slowness of some of these things. There's other techniques you can use.
- 12:47
You can use caching, you can use other components. Um, but if you are slower and more expensive, you will not be used. If you are, uh, slower and cheaper, and you're mitigating some of the effects by leveraging something like, uh, a fancy UI spinner or something that keeps your users entertained as the inference is being calculated, uh,
- 13:05
you can, uh, still win. Now, uh, knowing your end customer, as I said, is very important. And then the other very important thing is the number of times I see people defining a tool called get current date is infuriating to me.
- 13:20
Like, it is literally, like, import time, time dot now. Y- you know, like, just it's a format string. Just throw it in the string. Like, y- you control the prompt.
- 13:30
Um, so, uh, the downside of putting some of that information very high up in the prompt is that your caching, uh, is not as effective. But if you can put some of that information at the bottom of the prompt after the instructions, you can often, uh, get very effective caching.
- 13:45
Um, then there is... Like, I, I, I used to say we should fine-tune, we should do these things. Uh, it turns out I was wrong. As the models have improved and gotten more and more powerful, uh, prompt engineering has proven unreasonably effective for us.
- 14:01
Like, far more effective than I would've predicted. Within, uh, Claude 3.7 to Claude 4, we saw zero regressions. From Claude 3.5 to 3.7, we did see regressions on certain things when we moved the exact same prompts over to some of our, uh, users and some of our evals.
- 14:16
But from 3.7 to 4, we got faster, better, cheaper, more optimized inference in virtually every use case. So it was like a drop-in replacement, and it was amazing. Um, and I'm hoping future versions will be the same.
- 14:30
Uh, I'm hoping we're-- the era of having to adjust your prompt every time a new model comes out is ending. Um, and then finally, it's very important to know your economics.
- 14:38
Like, is this inference gonna bankrupt my company? Um, if you think about some of the cost of, uh, uh, the, the Opus models, you, you know, it may not always be the best thing to run.
- 14:50
Okay. So just in the interest of time, this is another great slide. This is, uh, from Anthropic actually. And, uh, when we think about how to create our evals, the vibe check, the very first thing that you do when you try to create, um, a, uh, a, a test, that vibe check becomes your first eval.
- 15:11
And then you change the data and the stuff that you're sending in, and lo and behold, uh, twenty minutes later, you do have some form of eval set that you can begin running.
- 15:19
And then you can go for metrics. Now, metrics do not have to be a score like a BERT or a, a, a, you know, a benchmark score that is calculated.
- 15:27
They can just be a Boolean. It can just be true or false. Was this inference successful or not? Um, that is often easier than trying to assign a particular value and a particular score.
- 15:37
Uh, and then you just iterate, you know? Keep going. And like I said, speed matters, but UX matters more. You know, this UX orchestration, prompt management, all of this great stuff, uh, is why we end up doing better than, uh, some of our competitors.
- 15:52
And then, you know, one of our customers, CloudZero, uh, we originally built a chatbot for them for you to chat with your AWS infrastructure and get cost out of that AWS infrastructure.
- 16:03
Um, we are now using generative UI in order to render, uh, the information that is shown in those charts. So in, just in time, we will craft a React component and inject it into the, uh, the rendering of the response, and then we can cache those, uh, components and describe in the prompt, "Hey, I made this for
- 16:23
this other user. Maybe it's helpful one day, uh, for some other user's query." And so this generative UI allows the tool to constantly evolve and personalize to the individual end user.
- 16:33
Um, this is an extremely powerful paradigm that is finally fast enough with some of these, uh, models and their lightning fast inference speed. Um, nature footage, we covered that earlier.
- 16:44
Uh, there's also knowing your end user, which is we had a customer, uh, that had users in remote areas, and so we would give, uh, text summaries of these PDFs and manuals and things, and that would, uh,
- 16:59
be great, and then they would get the PDF, and it would be two hundred megabytes, you know? And then, so what we found is on the back end on the server, we could take a screenshot essentially of the PDF and just send that one page so that even when they were in low connectivity areas, we could still
- 17:11
send the text summary of the full documentation and instructions, but just send the relevant parts of the PDF without them having to download a two hundred megabyte thing. So that's know your end customer.
- 17:21
We worked with a hospital system, for instance, that, uh, we originally built a voice bot for these nurses. Uh, and it turns out nurses hate voice bots because hospitals are loud and noisy, and the voice transcription is not very good, and you just hear other people yelling, and they preferred a regular old chat interface.
- 17:35
So we had to know our end customers, figure out what exactly they were doing day to day.
- 17:40
And then let the computer do what the computer's good at. Don't do math in an LLM. It is the most expensive possible way of doing math. Um, let the, the computer do its calculations.
- 17:53
And then prompt engineering, I'm not gonna break this down. I'm sure you've seen hundreds of talks over the last two days about the, uh, way to engineer your prompts and everything.
- 18:03
Uh, but one of the things that we like to do, uh, as part of our optimization is to think about the output tokens and the costs that are associated there and how we can make that perform better.
- 18:14
And then finally, know your economics. There's lots of great tools. There's things like prompt caching. There's things like tool usage and batch. Um, batch on Bedrock is a fifty percent off whatever model inference you're trying to make across the board.
- 18:26
And then context management. You can optimize your context. You can figure out what is the minimum viable context in order to get the correct inference, and how can I optimize that context over time.
- 18:36
And this, again, requires knowing your end user, knowing what they're doing, and injecting that information into the model. And also optimizing stuff that is irrelevant and taking it out of the context so that the model has less to reason over.
- 18:48
If you were interested in this and you want to learn more, if you want to talk more, um, I'm always happy to hop on the phone with customers. You can scan this QR code.
- 18:57
We like building cool stuff. Uh, I got a whole bunch of talented engineers who are just excited to go out and build things for customers. So if you have a super cool use case, come at me.
- 19:08
All right. Thank you very much. [upbeat music]