AI Engineer World's Fair 2024
Understanding AI Stakes to Break Production Code
Read the talk
Getting AI to Production Means Deciding What Can Go Wrong
Production AI requires more than a convincing prototype: the consequences of error determine where reasoning belongs, how systems are evaluated, and what users must be able to inspect.
From a talk by Philip Rathle
Before you start: Familiarity with language models, vector search, and basic application pipelines will help; the Python example uses only the standard library.
Why does an impressive prototype stall?
What changes between an impressive AI demonstration and an application people can safely depend on? The answer starts with application stakes: what happens when the system is wrong. Philip Rathle opens this production roundtable by asking participants to compare the hurdles they have solved with those they have not. The discussion operates under the Chatham House Rule, with audience contributions kept anonymous. Its organizing premise is that higher stakes bring higher acceptance bars—and different engineering obstacles.
Generative AI initially looks like a hockey stick: capability rises quickly, and compelling prototypes are easy to build. Actual projects encounter a production barrier. Adding retrieval-augmented generation, or RAG, with vector search can move some applications across it. Others still fall short.
Rathle presents knowledge graphs and GraphRAG as a further step in Neo4j’s approach. But even this progression is better understood as successive S curves than as unlimited acceleration: a technique unlocks a class of applications, then encounters another limit. The practical question is which barrier matters for the application being built.
| Application position | Example or consequence | What changes the stakes |
|---|---|---|
| Lower stakes | Bedtime stories; some text summaries | Several answers may be acceptable |
| Intermediate | A business copilot | Human oversight mitigates consequential errors |
| Higher stakes | An autonomous pilot | Errors can affect money, reputation, health, or safety |
These are positions on a spectrum, not fixed labels for entire product categories. Regulation, fines, and bias also raise the consequences of failure. A human in the loop can move a consequential application toward the middle: the system assists a decision instead of owning it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An apple, a vector, and a knowledge graph
Higher-stakes applications often need enterprise-specific facts and up-to-the-moment information that a model cannot reliably supply from its parameters. Retrieval brings that information into the answer-generation process. It can also reduce hallucinations, but Rathle’s metaphor of repeatedly loading the dice captures the remaining problem: improving the odds does not establish certainty.
The acceptable residual risk depends on the task. Rathle asks how tightly to fasten bolts on a 737 MAX 9. In Rathle’s hypothetical aircraft-maintenance example, even a 0.001% chance of error is unacceptable. That figure expresses his safety concern, rather than a measured model error rate or an established aviation requirement.
Now consider an apple. A person’s concept of it combines appearance, taste, texture, and other associations. A vector represents it as coordinates. Cosine similarity or spatial distance can use those coordinates to retrieve related fruit; that is a useful operation, but it is not the whole concept of an apple. Rathle characterizes LLMs and vector RAG as statistical ways of approximating the world through patterns such as word proximity and frequency.
His right-brain analogy describes the resulting behavior: creative, often right, occasionally very wrong, and difficult to explain. Knowledge graphs provide the complementary representation. Explicit structure makes information understandable to people and available for machine reasoning—the left-brain side of his analogy. These are metaphors for system behavior, not a neurological account of language models.
Structured knowledge and language generation have different jobs. A graph cannot do everything an LLM can. Combined with an LLM, however, it can supply facts, relationships that support reasoning and discernment, and persistent information that serves as long-term memory. The proposal is to combine those capabilities rather than expect either representation to solve every problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put exact reasoning in the right component
The discussion turns to a practical question: which lesson would have saved time if it had been learned sooner? Rathle’s first answer is to understand the stakes early without letting planning stop iteration. Start building, but let the consequences of failure influence tool selection.
His term for the next decision is locus of reasoning. If a question has an exact answer, a database or another reasoning engine may be the right place to compute it. The LLM can handle the language on either side:
- Translate the user’s question into a query.
- Have the database or reasoning engine compute the result.
- Express that result in language appropriate to the audience.
Deterministic execution does not make the whole pipeline correct: the generated query must still express the intended question, and the final response must preserve the result. But it moves the exact computation into a component designed for that work.
Audience adaptation then becomes a separate task from determining the answer. A response to a regulator should be serious and joke-free. An outbound marketing message might use different language for a high-school student and a retiree. The facts need not change when the presentation does.
An audience member offers a concrete version of this separation: instead of pasting Excel data into an LLM and asking it to identify an outlier, ask it to write Python that finds the outlier. The resulting computation can run without keeping the LLM in the execution loop. The model’s ability to write a procedure can be more useful than its ability to answer the numerical question directly.
For a small illustrative dataset, an explicit interquartile-range rule makes that division of labor visible. The values and rule below are a worked example of the suggested approach:
python
from statistics import quantiles
values = [10, 11, 12, 12, 13, 14, 50]
q1, _, q3 = quantiles(values, n=4, method="inclusive")
iqr = q3 - q1
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr
outliers = [
value
for value in values
if value < lower_bound or value > upper_bound
]
print(outliers) # [50]
Here the program applies a specified definition of an outlier. Choosing whether that definition fits the data remains a separate decision; the benefit is that the calculation itself is inspectable and repeatable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Semantic search still meets keyword habits
A team can build good retrieval and still misunderstand how people will use it. One participant describes investing in RAG, indexing, and semantic search, then discovering in production that customers continued entering keywords. The team had expected natural-language questions; users brought the habits of a familiar search box.
That mismatch required work to interpret or rewrite the input: was it a keyword query, or a question? The team had spent heavily on making search work before asking whether this was how customers wanted to search. Retrieval quality alone did not resolve the interaction problem.
Rathle asks whether a curriculum could help users discover that they can ask more than keyword queries. The participant acknowledges that onboarding had initially received little attention: the assumption was that everyone would simply start asking questions. Easing users into the new behavior became part of the product work, alongside making the underlying search function well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose projects that can meet the acceptance bar
Another production lesson is to kill more projects during experimentation. Running many experiments is useful only if the team eventually concentrates effort on the few that can meet users’ expectations. Some projects will not reach the required accuracy even with substantial additional work. The participant would rather focus on one or two that can make it all the way to production.
Rathle raises both achievability and business value as selection criteria. The response brings the discussion back to the user’s acceptance bar. A participant uses 90–95% accuracy as an illustrative level that may be insufficient for one use case, while 90% may be acceptable for another. No task, dataset, or accuracy definition accompanies those figures; they illustrate why expensive commercialization should follow a match between achievable performance and user needs.
A brief contribution sharpens the same point: expect less and use more common sense. That does not mean abandoning ambition. It means avoiding a product plan that depends on capabilities the system has not demonstrated.
The economic version of the problem begins with a customer asking for the best model at the fastest possible speed. Those are not sufficient requirements. Cost–quality and cost–speed trade-offs depend on what the application actually needs to do and how quickly it needs to respond. Without that understanding, a deployment can spend far more than necessary.
A small laboratory question set naturally encourages maximizing accuracy; production exposes the cost of doing so repeatedly. Rathle imagines a deployment consuming a year’s budget in a day to illustrate the danger of ignoring scale. His suggested response includes smaller models, with the choice driven by required quality and latency rather than by an unconditional demand for the strongest model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ship now without freezing the model stack
Rapid model improvement complicates the investment decision. The moderator tentatively recalls Bloomberg training a financial-data model, perhaps based on Llama 2, that performed well in evaluations. The moderator recalls Llama 3 outperforming that financial fine-tune, but the model identity and comparison are unverified. The anecdote raises a useful question without settling it: how much work will a subsequent base-model release make unnecessary?
Waiting is not a complete product strategy either. Enterprises want something working today, and teams want the competitive advantage of shipping. Rathle initially frames the wait in years; participants push back that meaningful advances are arriving on the scale of weeks. The engineering problem is to deliver value while retaining the ability to benefit from those advances.
One participant’s approach is to establish the full path through the system first, then improve its stages in parallel. Deliver an end-to-end result for the customer. If the first data-engineering step will not scale, assign work to improve it. If acceptable model performance currently requires an expensive option such as Claude 3.5 Sonnet, keep that stage replaceable as model capabilities and economics change. These model names describe the choices discussed at the time of the recording.
A working pipeline makes component improvements usable. Once the steps and their boundaries are understood, better components can be substituted into the path rather than requiring the product to be reinvented. Rathle connects this to the rising capability curves from the opening: improvements become valuable when the system has somewhere to put them.
That approach has a limit the participant has not solved. More agentic behavior makes orchestration itself a moving target. The team is looking for better tools, and it has already had to slow down because too many layers were changing at once. Interfaces and contracts between layers are necessary so that improving one stage does not unexpectedly break a downstream stage. The participant presents this as ongoing work, not a finished orchestration solution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validate capability before optimizing it away
The next question is how long an optimization should take. If a latency improvement requires months of engineering, a new model may arrive before the work finishes. A participant proposes putting a ceiling on implementation time, without specifying a universal number of months. Rathle’s two-year-project example pushes the concern to an extreme: a long project aimed at a temporary model limitation may become obsolete before it delivers value.
A different participant warns about optimizing too early. Teams that begin with a small fine-tuned model can incorrectly conclude that a task is impossible. Starting with a more capable model helps distinguish a genuine product or capability problem from a limitation introduced by the initial cost constraint. Production latency still matters, but it should not prevent the team from first learning whether the desired behavior is achievable.
The resulting sequence is:
- Validate the product need and capability. Establish whether a strong model can perform the task well enough.
- Account for proprietary information. Determine whether missing domain data calls for retrieval or fine-tuning.
- Optimize the working behavior. Move toward faster, cheaper models, potentially using larger-model outputs as fine-tuning data, while checking that accuracy holds.
Rathle names GPT-4o and Claude 3.5 as starting points for that feasibility check. His heuristic is that if the stronger model cannot do the task, a smaller model probably will not—while acknowledging that highly proprietary data changes what the model must be given.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation starts with the data you will need
Consider a pipeline in which an LLM converts unstructured material into structured data for a traditional machine-learning model. The application needs a way to determine whether that conversion is good enough and whether subsequent changes improve it. A participant’s lesson is to build the evaluation pipeline from the start—and first ask whether the necessary evaluation data exists.
If suitable data is missing, begin generating it on day zero. Otherwise, a team may reach the point of having an LLM embedded in its application without being able to tell whether a prompt change made the system better or worse. Designing evaluation only then creates a long delay: the application exists, but the evidence needed to guide its development does not.
Rathle places this alongside the usual operational framework. DevOps, evaluation, and data quality are connected responsibilities. The ability to deploy a change is not the ability to judge it; the evaluation framework, in turn, depends on having the right data at sufficient quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make generated answers recognizable and inspectable
One team deliberately avoids hiding AI behind an apparently ordinary interface. A participant describes gradually displaying generated responses as a cue for users to be vigilant, along with distinct colors and icons that identify AI interactions. The intent is to make the nature of the output visible before users treat it as authoritative.
The interface also gives users tools to inspect an answer: what sources were used, what inputs went into it, and what might have gone wrong. The team does not simply abstract generation away and hope for the best. Rathle connects visible generation to moderated expectations: seeing an answer emerge can remind a user that it deserves consideration rather than automatic acceptance. These are the team’s reported interface choices; the discussion does not establish how much they change user vigilance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Human review needs more than a citation list
Explainability becomes especially concrete when the output could influence medical treatment or equipment maintenance. In response to Rathle’s question about regulation, a behavioral-health practitioner describes the team’s current operating principle: always retain a human in the loop. They are building a copilot. The participant’s remark about never becoming a pilot is offered as reassurance to others, rather than as a firm technical prediction.
For that human review to mean something, the reviewer needs access to the inputs that produced the output. Recommendations and selected case information must be traceable to root data. Rathle asks whether showing inputs is enough to earn trust, then points out the practical burden of references: users may not read three 50-page papers just to check an answer. Evidence must be available in a form that fits the review task.
The practitioner describes two review arrangements:
- Knowledge lookup: show the retrieved source documentation beside the answer, with citations.
- Therapy-session summary: show the collected session data beside the generated summary, paragraph by paragraph.
For summaries, the user can change and edit the resulting paragraphs. The system supports a reviewable draft; the final summary becomes the user’s work rather than an untouchable model artifact.
Rathle then asks whether human judgments feed back into the producer through RLHF. The practitioner describes several checks and feedback channels:
- A separate, quick evaluation stage checks whether the output matches the inputs, including a safety-oriented check.
- User edits and thumbs-up or thumbs-down responses provide feedback.
- The team performs its own clinical evaluation of outputs.
This describes layered evaluation and feedback collection, not a specified RLHF training procedure. The team inserts checks wherever it needs additional confidence in what the system is producing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a human reviewer cannot be included
The final example tests the limit of human oversight. A participant working on regulated personal-loan recommendations cannot include a human reviewer in the workflow. Instead of declaring a single loan best, the team gives the end user three options with explanations. The participant describes this as less deterministic in the sense that the product presents choices rather than one definitive recommendation, and reports that it helps address regulatory concerns; the discussion does not establish compliance with any particular rule.
Rathle checks who receives the options: an intermediary reviewer or the end user? They go directly to the end user because intermediary review is unavailable. The participant adds that the selection uses logic beyond raw LLM output, although the logic itself is not specified. The final design therefore changes the decision the AI is allowed to make: it supplies explained alternatives for the user to consider, rather than presenting one generated answer as the decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official explanation of the rule permitting information sharing while protecting participant identities and affiliations.
Further reading
- The GraphRAG manifestoArticle
Philip Rathle explains graph-enhanced retrieval, contrasts vector and graph representations, and develops his argument for enterprise GraphRAG.
Foundational research combining pretrained language generation with retrieved Wikipedia passages.
Bloomberg's original account of its financial language model, training data, and evaluation methodology.
Updates since the talk
Current examples for vector search, hybrid retrieval, graph traversal, and natural-language-to-Cypher pipelines.
Read the complete timestamped transcript
- 0:00
[upbeat music] So the idea here is a lot of you are solving some of the most leading-edge AI, GenAI problems in the world in, in, in this room, and then a lot of you are trying ambitious things.
- 0:23
So main goal was to get folks, um, learning from each other. So think of this as a logical roundtable, uh, not, not very round physically. Um, so come away with insights about specifically your getting to production challenges by learning from each other.
- 0:40
So two rules of this AI roundtable. One is we're all-- we'll all be stronger if we figure this out together. Number two is Chatham House Rules. Um, that's for Peter's benefit.
- 0:51
Um, for us [REDACTED:origin] in the room, what happens in Nob Hill stays in Nob Hill.
- 0:57
Um, so l-let me actually kick it off by sharing some of my personal learnings. I'll go through maybe f- uh, five, five minutes or so of what I've learned in relation to getting to production, but specifically around the topic of stakes.
- 1:11
Um, and by stakes I mean, like, how high stakes is the problem that you're trying to solve. And I found there's a relationship between the bars and the obstacles that you run into as the stakes increase.
- 1:21
So I'll just share my own journey, and hopefully this is useful and food for thought. But as I'm going through it, think about what you're experiencing and the hurdles, 'cause what I'm gonna ask after I go through my own section is what, uh, what are some of the hurdles you are running into?
- 1:38
What are some of the ones that you haven't solved? What are some of the ones you have solved? So that we can learn from each other.
- 1:45
All right, so it all started like this. GenAI is a h-- GenAI is a hockey stick, right? Um, and then we kind of ended up with actual project experience where you can do, and I think Lucas said this earlier, lots of amazing prototypes.
- 1:58
It's so easy to build an amazing prototype, but then getting to production is harder. You have a higher bar. And so what a lot of folks have figured out is you can use RAG with vectors and all right, now it looks like a hockey stick.
- 2:12
And in some cases it does. In some cases you can get to production with this. But then there's still a class of use cases where that's not enough. And so what, you know, you'll hear from the Neo4j folks here, and if you go to the booth and attend Emil's talk and some of the other ones later, is
- 2:28
there's another unlock here with knowledge graphs and with GraphRAG. Um, again, like, will this be a hockey stick? I think the world moves in S curves, so this is probably more realistic and there's probably gonna be some other thing.
- 2:40
I don't know what that looks like. Um, but why this pattern? Um, so I-I've come to see a spectrum of application stakes, and it's a whole spectrum, right? But to kind of paint opposite ends and maybe something in the middle, if something is low stakes, then maybe it's more business peripheral.
- 2:59
Um, it's things like summarizing text. Here I have someone, you know, a children's bedtime story. Um, there's no single right answer. Uh, the stakes are a bit lower. Um, and then on the high end of things, like a wrong answer can have significant impact in terms of dollar value, brand or reputational impact, health and human safety, regulation,
- 3:23
fines, bias, all that stuff. And then, um, and then there's this in-between zone where, you know, it might be higher business stakes, but I've got some mitigations. Like, I've got a human in the loop.
- 3:35
Um, so, uh, so here you have a difference between at the far right, um, uh, pilot and in the middle, co-pilot. Uh, so there's kind of an interesting relationship between pilot, co-pilot, and, and stakes.
- 3:51
And, you know, the problem with high-stakes use cases is you bump into the meme, right? Which you've all seen. Um, and specifically it's things like needing enterprise domain knowledge, up-to-the-moment knowledge.
- 4:02
This is what RAG solves. Um, solving for h-hallucinations. Again, RAG solve this and depending on how you solve it, you're, you're loading the dice, loading the dice, loading the dice.
- 4:13
And then for-- depending on the stakes, your tolerance for error, you know, could be, you know, potentially gets very high, right? How t-- how tight should I tighten my bolts on my 737 MAX 9?
- 4:24
Like, let's not have a 0.001% chance of error, 'cause that's not good enough. Um, cool. So let's look at how we're representing data. So here's an apple, and think about how we conceptualize an apple as a human.
- 4:39
It's, you know, it's this multidimensional, multisensory kinda thing. Um, now let's look at a vector view of an apple. It looks something like this. [chuckles] Um, so I can find other fruit by doing a, you know, cosine similarity or spatial distance, um, operation with, uh,
- 5:01
w-with this representation of an apple. It's useful. Again, good for certain things, good-- not, not as good for other things. And if I characterize the behavior that you get with LLMs generally and with vector-based RAG, it's this, you know, kinda world of statistically computing, uh, the real world based on, um, word proximity statistics and word frequency statistics.
- 5:25
So you get this behavior that's kind of a little bit like the right brain. You know, it's creative, it's mostly right, sometimes really, really wrong. I don't really know why it's wrong, it just does its thing.
- 5:35
Uh, a bit of a black box. Enter knowledge graphs as a way to bump up the, the stakes on things. So this gives you a, um, more structured representation that is understandable by humans, can be reasoned upon by machines, and this gives you a bit more of left brain behavior.
- 5:54
So it can't do all the things that LLMs can by any means, but when you combine it with an LLM, you, you-- being able to bring in facts, reasoning, discernment, long-term memory, um, can create a better together.
- 6:08
So that's my quick- Um, sharing around stakes and unlocking things in different levels. So let me turn it over to folks in the room, and Peter will be coming around with a microphone.
- 6:21
So let me start with the question: Does anyone have... What's one learning that would've saved you a lot of time if you'd known it sooner?
- 6:33
Come on, all at once.
- 6:34
Anyone, just raise your hand.
- 6:36
Yeah.
- 6:36
Blurt it out if you're-
- 6:37
Well, maybe if, if no one's got one-
- 6:39
Yeah
- 6:39
... maybe you have one, like, just off the cuff. It might inspire other people to have a story-
- 6:45
Um-
- 6:45
... get the scope.
- 6:46
I mean, it, it comes back to stakes and, you know, un- understanding what those are upfront, um, for different applications. So w- we've heard a lot today about iteration and how important that is.
- 6:58
So definitely iterate. You know, don't overthink things initially. But, you know, to also think and, think and plan ahead, and that can, that can guide things like what tools you use.
- 7:09
Also locus of reasoning is, is something I think about, is if, if there's something-- if there's a question that has an exact answer and it's a der- deterministic question, then, you know, maybe you can use the LLM, which is inherently non-deterministic, to complement some other technology that can give you a deterministic answer, which you have.
- 7:28
You've got databases, um, and, and other kinds of reasoning engines. And then use the LLM to do what's, it's exceptionally good at, which is maybe take the question, uh, turn it into, um, a query.
- 7:41
Take that query, formulate it back into some text, and then maybe modulate that text depending on who the audience is. If it's going to regulator, then very serious. No jokes.
- 7:50
If it's going to, you know, a, an outbound marketing email, then it can, uh, maybe go to, uh, be phrased differently depending on who the target audience is. Is it someone in high school?
- 8:01
Is it a retiree? Um, yes, I see a hand, though-
- 8:04
Oh
- 8:04
... though over there.
- 8:04
Cool. Let's go. [laughs] Here we go.
- 8:07
Going around.
- 8:08
Ah, so I'll go here.
- 8:10
Um, I've, uh... One, one learning that I wish I had had before is instead of asking the LLM to solve a problem, right, like paste in Excel and say which one's the outlier, just ask it to write the code to solve that problem, which it can easily do, and then you can leave the LLM out of the
- 8:27
loop. Um, right. You say, "Find the outlier." It will do some Python outlier machine learning thing, and then you have the answer.
- 8:35
Cool. Yeah. So the LLM might not be good at certain kinds of reasoning, but it can write code that will do the reasoning, which is a pretty amazing hack, actually.
- 8:43
Thanks. Um, I think you were-
- 8:45
Oh, everyone's getting involved now
- 8:46
... I know. I'll, you, you, you work out who's next.
- 8:48
I'll... Wait, I'm sorry. Who had their hands up? Oh, let's go here.
- 8:52
Yeah. So one of the things we learned was, um, when we were kind of getting excited about the RAG and indexing and doing the semantic search, and when we put it to production, that our customers were not used to doing queries like this.
- 9:07
They always resorted back to keywords. It's almost like they were-
- 9:11
Mm-hmm
- 9:12
... like-
- 9:12
Like search
- 9:12
... trained to do this.
- 9:14
Yeah.
- 9:14
And so getting them along, uh, and so we had to write, rewrite, like is this question actually a query for keywords or is it like the, like a query they ask like a, a question?
- 9:26
So that was a learning that, you know, we, we made, uh, while putting it into production. And, uh, we put a lot of effort in making the search well, [laughs] but we, we, we didn't ask upfront like, well, is this actually how you wanna use it?
- 9:39
And so that was one learning there.
- 9:41
And so did you have a sort of curriculum to help people understand how to ask questions and that they can do more than just keywords?
- 9:49
Uh, so yeah, we, we didn't put attention to this. We were just-
- 9:52
Yeah
- 9:52
... assuming like, yeah, everybody's now gonna use like-
- 9:54
Yeah. Yeah
- 9:54
... putting questions in. So that was, you know, the surprise and the learning there, like-
- 9:58
Yeah, that's cool. Yeah
- 9:58
... we needed to ease them in.
- 9:59
You need to understand what the technology's capable of, and they can do more than just-
- 10:02
Yeah. But it's like the-
- 10:03
... what you're used to. Like it's not a Google search box
- 10:03
... adapt the user to how the new thing is supposed to work, uh-
- 10:07
Yeah
- 10:07
... in a way. Like that was a learning.
- 10:09
Cool.
- 10:10
Thanks.
- 10:13
Um, I think the lesson is kill more projects than experimentation. So right now, I think one of the big lessons we learned is we did a lot more experimentation, and r- if I have to look back, I will focus on few that will go all the way to production.
- 10:29
Yeah.
- 10:30
Because a lot of these experimentation, even with the best effort, will never meet the user expectation on accuracy. So if we can kill and... a lot of projects, but focus on one or two that can, we can take all the way to production is probably where I will focus a lot more.
- 10:46
And people spend too much time, uh, on not being focused.
- 10:50
And when, when deciding what project to focus on, I imagine there's an element of how achievable is it and how valuable is it. H-how, how do you-
- 10:57
The thing is for us, like it, it... Can we get to the level of accuracy that the user is expecting?
- 11:03
Yeah.
- 11:03
Commercialization is expensive, but we can do it to 90, 95. That's not good enough. Why are we spending time on it now? But work on the use cases that will go to them, where 90% is good enough.
- 11:15
Yeah. So matching up with user expectation of accuracy, and then those projects are bound to succeed and go with those. Cool. Thanks.
- 11:23
Okay.
- 11:23
My learning is, uh, expect less and use more common sense.
- 11:28
Yes.
- 11:28
It's a funny one.
- 11:29
Yeah. I like that.
- 11:37
Thank you.
- 11:38
One learning that I've experienced from our customers when they're building deployments is, you know, it's really easy to walk in and say, "I want the best model, and I want it to run as fast as possible."
- 11:50
But when you're making, you know, actual cost to quality trade-offs and cost to speed trade-offs, you know, you can, you can really overspend versus what you really need if you don't have a great understanding of what you're actually trying to build, how fast it actually needs to run, that kind of stuff.
- 12:08
Yeah. Yeah. That in, in a lab with a small number of questions, you're naturally gonna go for ... the best accuracy, but then enter production if it, you know, if, if, if you're burning a year's budget within a day, which is very possible with this stuff, then maybe you need smaller models or multiple...
- 12:24
Yeah. Gr- great one.
- 12:26
Actually, I just had one pop up in my own mind, um, that I remember seeing, which was when Bloomberg were, uh, training, I think it was Llama 2 at the time-
- 12:36
Mm
- 12:36
... to handle a lot of the financial data that they had, and it passed really well on all the evals and everything. And then [laughs] they found out when Llama 3 came out, it actually outperformed their fine-tuned model they'd spent all this time producing.
- 12:47
Um, and so I'm not entirely sure what the takeaway from that is, but it's almost this idea that if you just wait for the models to get better, then you kind of save work.
- 12:56
But then I guess that's not necessarily gonna fly in the enterprise where [laughs] it's like, "We want something working today."
- 13:00
Well, th- this is a great... I'd love to hear what people in the room are experiencing, 'cause there's this real tension of, all right, if I just wait a couple years, like, this stuff is all gonna figure itself out and it'll be way better, but who wants to wait a couple year?
- 13:12
Like, we, we wanna get out and get competitive advantage. [laughs]
- 13:16
Not a couple years, right? Like, we're seeing crazy advances.
- 13:18
Yeah, yeah. You wait, you wait a few [laughs] weeks.
- 13:20
So like, we're, we're just focused on, like, the full path and getting it end to end, and then, like, in parallel tracks, optimizing every part of that, that journey, right?
- 13:29
Yeah.
- 13:29
So trying to end the end shipping customer value, and then, oh, our data engineering part of that was like a clutch and that's not gonna scale, so let's get people working on that.
- 13:37
And it's like an interface between every step of this chain that we're putting together. And so, like, if I encounter an issue where my LLMs aren't performing and it's like, "Oh, no, we gotta use, like, 3.5 Sonnet and it's gonna cost us a lot," well, just wait, you know?
- 13:50
It's gonna come down a little bit.
- 13:52
Yeah.
- 13:52
And then kind of just keep optimizing as these improvements happen across that stack.
- 13:56
Yeah, yeah. The, the, the bars I was showing earlier are naturally all rising, like, across all those dimensions, and if you understand what the pipeline looks like, then you can just drop stuff in, right?
- 14:08
Figure out your pipeline first-
- 14:09
Yeah
- 14:09
... and then you can just iterate on each one of those components.
- 14:12
Yeah.
- 14:12
That's, that's what's been working really well for us.
- 14:14
Cool. I like... So o- o- o- one question to drill into that is, I can see pipelines getting more complex with more agentic stuff and so on. Like, how do you rec- like, that one is a moving target too.
- 14:26
How do you see handling-
- 14:27
It's, it's-
- 14:27
How are you handling that?
- 14:28
It's not something we've figured out yet. Um, obviously looking for tool, better tools to help orchestrate that.
- 14:34
Yeah.
- 14:34
We know we are making improvements on various parts of this thing, but, like, I've had to slow down the team. Like, we're making too many changes at once now across the different things.
- 14:42
Mm-hmm.
- 14:43
So it sounds like we have to slow down a little bit with interfaces and contracts between the layers so that we can iterate on one side and not affect the downstream on another.
- 14:50
So I don't know that yet.
- 14:52
Cool. Great. Anyone else on this question?
- 14:54
Just, just to piggyback.
- 14:55
Yes.
- 14:55
Um, when you're thinking of the latency improvements you can get if you have a solution that might take you months to implement, you have to evaluate that with the trade-off and say, "By the time we finish this, will a new model have come out?"
- 15:08
Yes.
- 15:08
So I think the time it takes to implement new solutions, there should be, like, a max, like a ceiling. Like, if it takes more than however many months, you're just wasting your time because it's gonna be throwaway.
- 15:18
I love that. Yeah. If it's a two-year project, then you're-
- 15:21
Yeah
- 15:21
... w- why bother? Like anything-
- 15:22
Basically-
- 15:23
... you do now is gonna be irrelevant.
- 15:24
Right. I need another solution.
- 15:25
Yeah.
- 15:25
Basically.
- 15:26
Yeah. Love it.
- 15:30
I'm gonna dash over here.
- 15:31
I could also just shout.
- 15:32
Yeah. It sometimes works. But, um- [laughs]
- 15:35
Thank you. Yeah, one, I've-- all of that I really resonate with, and one thing that we've seen is companies that have succeeded have avoided premature optimization. So obviously you want to be sort of aware of latency in production and, you know, make something that'll be realistic.
- 15:50
But the teams that have sort of started with that and gone, "We're gonna use a small fine-tune model," often incorrectly conclude that a task isn't possible. And also there...
- 15:59
people are often able to take data from a larger model and fine-tune it later. And so I always think of, like, moving down the model size stack or fine-tuning as optimizations, and you want to validate the product need and whether it works, whether it's possible, usually with a better model first.
- 16:14
At least we've seen that be much more successful.
- 16:16
That's a great... I, I've, I've heard that too, like start with GPT-4.0 or I guess the, now, now maybe Claude 3.5 and, um, if that can't do it, then the smaller models probably won't.
- 16:28
And, and I guess you have an exception for data that's highly proprietary, and that's, that's where fine-tuning and RAG come in.
- 16:34
Yeah. We do the same thing. Just optimize the best out of the box, and then we kind of slowly bring it down-
- 16:40
Yeah
- 16:40
... to more performant, cheaper models, hopefully trying to get the same accuracy out.
- 16:46
Cool.
- 16:46
I think building up on, on something that you mentioned as well earlier when, for example, you're using a, a, a large language model to turn unstructured data to structured data to then feed it into, like, traditional machine learning models.
- 16:56
Mm-hmm.
- 16:57
One of the key learnings for us was if we're, um, if we're building our eval pipeline from the get-go, which, uh, is really starting to be echoed, you know, trying to figure out what your eval should be, start, get it, get it ready from the start, is to understand do you have the right data to build your
- 17:12
eval pipeline? And if not, start from day zero to make sure that you generate that data for your eval pipeline. Because if you leave it to stay too long and then realize, oh, we've, we've built an LLM model and it's in our app, but we cannot evaluate whether, uh, the fine-tuning on the prompts makes it better or
- 17:31
worse, you're then spending a lot of sort of long leg time on, on figuring out what your eval pipeline should be. So our key learning was really start, as, as you start building your app, also start with your eval pipeline.
- 17:43
Make sure that you have the data that's there.
- 17:45
Cool. That, that's maybe a corollary of the, okay, you, you need your DevOps kind of framework, but then you need your eval framework, and then your data framework is closely related to that, is making sure you have the right data.
- 17:59
Yeah, and the quality's there.
- 18:06
Um, one of the things we learned is not hiding AI from the end user. So we're- Like explicitly, for example, generating the responses slowly, so as a hint for the user-
- 18:19
Hmm
- 18:19
... that they, they should be more vili- vigilant. We're using specific colors in the UI to make it clear, uh, the icons that they click on. So kind of make sure that they understand that there's a special thing they need to do.
- 18:32
And then when they wanna check, we give them as much tools for them to validate the answer that was generated to see what, what went wrong, what were the sources, what kind of, uh, went in there.
- 18:43
So that's one of the lessons that, uh, not to abstract this away and kind of hope for the best. But, uh-
- 18:48
This is related to your moderate expectations point.
- 18:50
Yeah.
- 18:50
If the user can actually see the thing coming out, then that's a trigger that, um, all right, it's not instant, it's coming from some machine, I need to consider-
- 18:59
Yeah
- 18:59
... things. Cool. I have, um, I wanna ask if anyone,
- 19:05
um, dealing with regulation or something where explainability is a big issue, and if so, how are you solving that?
- 19:17
What sort of industry are we talking? Like, it, like a medical something or-
- 19:19
Um, yeah, I, I could think about medical, like if you're trying to pre- if, if the end game is prescribe something or provide some treatment.
- 19:26
Yeah.
- 19:26
I can think about, um, equipment maintenance is another one.
- 19:29
We work in the behavioral health space, um, and a lot of their... There's a lot of regula- regulation, and the one way we've determined to solve it as a kind of a core principle is just always human in the loop right now.
- 19:42
Yeah.
- 19:42
Like, we're only in the copilot stage. In fact, I tell people we'll never get to the pilot stage, but that's just to not scare them. But, uh-
- 19:49
Yeah
- 19:49
... we're, we're, we're very focused on copilot right now, uh, and making sure there's a human in the loop. Um, but as part of that we have to give transparency to the inputs that generated the outputs, so they can kinda see things side by side.
- 20:01
Um, if we make recommendations or we pick some data out that we think is interesting, uh, for the particular case, uh, that they're working with, um, they can, they can trace it back to the, the root data.
- 20:13
And so do you find the inputs are sufficient for getting the humans in the loop to trust the answer enough? 'Cause I, I... So you've got inputs. Another way to do it is here are the references that, you know-
- 20:27
Yeah
- 20:27
... but then they're not gonna read the three 50-page papers maybe.
- 20:30
We, we, we... So if it's, if it's doing a, a lookup, a knowledge lookup, we're showing the source documentation that we find based on their query, uh, so they can see it kinda side by side-
- 20:41
Yeah
- 20:41
... um, with those citations. Um, if it's doing a generation, uh, so let's say we're summarizing data that was collected in a therapy session, um, we show the data side by side with the actual summarization of that.
- 20:51
Uh, it's kinda like on a paragraph by paragraph basis so that the users have complete control of the, the resultant summary. They can change it, edit it. It's theirs at that point.
- 21:02
And then are you... Last question. Are you using the, um, judgment of the human in the loop as like [REDACTED:username] to improve the producer?
- 21:12
Uh, yeah, we run our own eval on any of our output with a separate, um, stage just to make sure that, like, did the input... Does the, does the output match kinda the inputs?
- 21:21
And that's just a quick one. It's also for a, a safety perspective. But then we use the, whether or not the user edited or, and/or just gave us thumbs up, thumbs down.
- 21:29
Yeah.
- 21:30
Uh, and then we do our own clinical evaluation at the end, uh, on our own, with our own team, just making sure, like we're just... Everywhere we need to, we're putting in that eval stage to make sure that we, we know what, uh...
- 21:43
we're comfortable with what's coming out.
- 21:45
Cool.
- 21:46
Uh, I'm just gonna go over here. Just one thing affects, uh, whether we can do one more or multiple more is, is our next speaker here? 'Cause I've not-
- 21:54
Yes.
- 21:55
Oh, fantastic. Cool. [laughs] Okay. Right, unfortunately this will have to be the last one then for now, but, uh, I assume this, this discussion-
- 22:01
Yeah
- 22:01
... can continue at lunchtime as well, 'cause it seems everyone, there's so many shared concerns, um, and stories that people have got in the room, so it would be a great thing for people to discuss at lunch.
- 22:11
Yeah.
- 22:11
Absolutely. Here we go.
- 22:13
I'll keep this short. So a hu- human in the loop is clearly a solution, but that's not always possible. So what we do is give options. So this is, uh, uh, in a, one of our use cases is personal loan recommendation.
- 22:27
Hmm.
- 22:27
It's regulated.
- 22:28
Yeah.
- 22:28
And then so we give, rather than, um, ever say, "This is the best loan for you," we give three options and give explanation for these, and so it becomes less deterministic and less, uh, and, and that, that's how we take care of the regulation.
- 22:42
That helps.
- 22:42
Okay, cool. Yeah. So g- give a few options, and then do you do that, do that directly to the end user or to the human in the loop?
- 22:47
It's end user because we cannot have human in the loop.
- 22:49
Right. Right.
- 22:49
But I think the-
- 22:50
Yeah
- 22:50
... the three options that we se- select are clearly t- show some more logic to it than just coming, that's coming out of the LLM.
- 22:58
Cool. Thanks. Well, let's definitely keep this one alive throughout the day as you're-
- 23:03
Thank you
- 23:03
... mingling with each other and lunch and so on. But, uh, thanks. I appreciate the participation. [upbeat music]