AI Engineer Summit 2025
Building Trust in Enterprise AI: Evaluating Domain-Specific LLMs for Real-World Financial Scenarios
Read the talk
When Financial Models Answer Beyond Their Evidence
FailSafe tests whether financial language models can handle damaged queries while recognizing when the supplied documents cannot support an answer.
From a talk by Waseem Alshikh
Before you start: Basic familiarity with language models and retrieval-augmented generation is helpful; no finance expertise is required.
Does better general accuracy make specialization unnecessary?
If general language models keep improving, is it still worth building models specifically for finance or medicine? For Waseem Alshikh, Writer’s cofounder and CTO, this is a practical investment question. He traces Writer’s model development back to its founding in 2020 and early encoder and decoder models. At the time of the talk, he estimates that the company has published roughly 16 models, with another 20 on the way. Its portfolio includes general models—Palmyra X, Palmyra 3 and Palmyra 4—alongside models for creative work, financial services and medicine, including Palmyra Medical. Palmyra 5 is still forthcoming in this account.
Alshikh describes good general models as reaching roughly 80–90% accuracy, without identifying the benchmarks behind that range. The improvement he has observed since early 2024 raises several alternatives to continued specialization: concentrate on general models, fine-tune them, or invest in reasoning models. If those approaches already deliver the needed performance, maintaining separate financial and medical models may be unnecessary. The decision requires evidence about what happens beyond ordinary benchmark questions.
Financial services supplies the test case. The same evaluation question applies to medical and customer-support models, but those domains need their own evidence. Alshikh mentions similar results from medical evaluation without presenting that evaluation here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Break the query without changing the underlying task
Writer develops FailSafe, documented in Expect the Unexpected: FailSafe Long Context QA for Finance, to test whether strong benchmark performance survives realistic financial requests. Its design separates failures in the user’s query from failures in the context supplied to answer it. That separation matters: recovering a user’s intended question and recognizing insufficient evidence require different behavior.
The first query perturbation is misspelling. Instead of assuming clean input, the evaluation introduces spelling mistakes, punctuation problems and common typing errors. The displayed example makes these defects concrete, highlighting split errors, segment errors, real-word errors and common typos in a financial query. A useful model should recover the intended question when the wording remains interpretable.
Two further query variants test how much the model depends on polished financial language:
- Incomplete queries: Remove keywords or other information, leaving parts of the request unclear.
- Out-of-domain queries: Express a financial question without expert terminology. Alshikh describes nonexpert wording and pasted general material; in the published benchmark, this perturbation uses everyday-language rephrasing.
Together, these variants test robustness to how people ask, rather than assuming every user knows the domain’s vocabulary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What if the uploaded document cannot answer the question?
The second family of tests changes the evidence available to the model. Missing context asks for information that the prompt does not provide; the published benchmark implements this by removing the document entirely. OCR errors preserve a document but damage its text during the simulated conversion from a physical page: characters become corrupted, spacing changes and words merge. These tests distinguish absent evidence from evidence that is present but difficult to read.
Irrelevant context introduces a particularly revealing failure: the user asks about one financial document but uploads a completely different document. Does the model identify the mismatch, or answer anyway? A fluent answer is not sufficient here. The model must determine whether the supplied evidence supports the requested claim.
A small Python fixture illustrates that distinction. Suppose the question asks for revenue in an annual report. The clean and OCR-damaged versions retain the relevant fact; the missing and irrelevant versions do not. The values below are a teaching example of the perturbation pattern:
python
question = "What revenue does the annual report report?"
cases = [
{
"id": "baseline",
"question": question,
"context": "Annual report: Revenue was $12 million.",
"expected": "answer_from_context",
},
{
"id": "ocr_error",
"question": question,
"context": "Annual report: Revenuewas $12 million.",
"expected": "answer_from_context",
},
{
"id": "missing_context",
"question": question,
"context": "",
"expected": "identify_insufficient_context",
},
{
"id": "irrelevant_context",
"question": question,
"context": "Office policy: Visitors must sign in at reception.",
"expected": "identify_insufficient_context",
},
]
The expected field specifies the desired behavior, not an observed model result. Keeping the question fixed makes the change in available evidence explicit.
The financial evaluation also needs diversity across its examples, rather than many variations of one narrow request. Alshikh shows a data and token inventory, then says that the white paper, data, evaluation set and leaderboard are openly available through GitHub and Hugging Face. The FailSafeQA dataset provides a public starting point for examining the cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Score recovery and grounding separately
The evaluation asks two questions: can the model produce the correct answer, and does it follow the supplied context? Robustness and context grounding are separate requirements. A model can recover from a typo while still failing to recognize that a document contains no answer.
The original paper’s methods give these dimensions more precise meanings:
| Dimension | What is evaluated | Aggregation |
|---|---|---|
| Robustness | Baseline and answerable perturbations | Average of each example’s minimum compliance |
| Context grounding | Missing or irrelevant context | Average compliance under those conditions |
Taking the minimum across answerable variants makes a fragile variant matter: success on the clean question does not erase failure when the same task arrives with a defect. Grounding instead tests appropriate behavior when the evidence is unavailable or irrelevant. Neither measure is unrestricted factual accuracy.
Alshikh presents a calculation overview and two lists of models, separating chat models from thinking models, before moving through the result charts. This comparison tests whether reasoning capability also brings better adherence to evidence, rather than treating those capabilities as interchangeable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Willingness to answer can hide a failure
The thinking models initially appear helpful because they keep answering. Under faulty context, that behavior becomes a liability. Alshikh reports that models continue producing answers even when given wrong data or a completely different grounding document, and associates that behavior with increased hallucination. The failure is not an inability to generate an answer; it is an inability to respect what the evidence permits.
On the answer-focused results, Alshikh reports that general and domain-specific models are close, with thinking models scoring slightly higher. The grounding results expose a different pattern, with text generation and question answering standing out as weak areas. Looking only at answer performance would conceal the cases where the model should recognize that the context is inadequate.
Moving from charts to the underlying numbers, he calls out o1, o3 and Palmyra Fin. Misspelled, incomplete and out-of-domain queries receive strong results: models can often handle poor grammar or imperfect wording and still recover an answer. That success establishes one useful capability, but the next part of the table asks whether they also stay within the attached evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More reasoning does not guarantee better grounding
Alshikh pauses on the grounding table and asks what looks different. A brief response—“Smaller is better”—gets his agreement. He describes larger thinking models as roughly 50–60% worse on grounding, but does not specify the comparison denominator. The useful conclusion is narrower than a general rule about model size: in these financial tests, some smaller models follow the supplied context better than larger thinking models.
The mechanism is visible in the behavior he describes: a context is attached, a question is asked, and the answer draws on information entirely outside that context. Irrelevant-context tests show the same problem. A model’s ability to supply information from elsewhere can therefore work against a task whose requirement is to answer from the attached document. The highlighted table marks selected context-grounding question-answering scores and points to Palmyra Fin.
This leads Alshikh to question whether the observed chain of thought deserves to be called thinking. His criticism concerns these domain-specific tasks: additional reasoning has not prevented unsupported answers. Robustness to damaged input, correctness of an answer and avoidance of hallucination can diverge, so progress on one does not settle the others.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reliability requires more than choosing a model
Even the strongest result leaves substantial room for improvement. Alshikh describes the best score across robustness and grounding as no higher than 81%. He translates that into roughly 20 completely wrong responses per 100 requests, but the published measure requires a more careful interpretation: the original paper reports Palmyra-Fin compliance of 0.81, combining robustness of 0.83 and grounding of 0.80. It is a composite score, not a measured production error rate.
That remaining gap motivates a full system for reliability. Alshikh calls for retrieval-augmented generation, grounding and guardrails around the model. Retrieval supplies evidence; grounding keeps answers tied to that evidence; guardrails provide constraints around the resulting behavior. The context-failure tests explain why merely attaching a document cannot be the whole solution: the attachment can be absent, damaged or wrong, and the system still has to respond appropriately.
The talk returns to the original investment question: should Writer continue building domain-specific models? Alshikh’s answer is yes, based on the implementations and financial benchmark results presented. General accuracy is improving, but reliable adherence to supplied context still lags. Specialized model development remains part of the response, alongside the retrieval, grounding and guardrails needed to make those models useful in real financial workflows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original benchmark paper with perturbation definitions, scoring equations, evaluation prompts and model results.
Public dataset containing 220 financial examples with reference answers, citations, query variations and OCR-corrupted contexts.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hello, everyone.
- 0:18
My name is Waseem. I'm one of the co-founder and CTO at Writer.
- 0:23
Today, I'm going to just tell you a quick story about actually why we're building at Writer, what we're doing. But before we dive in, I would love just to give you a quick, uh, history of Writer.
- 0:34
So Writer's, we started the company in twenty twenty. We love to say the story of Writer is a story of the transformer. We started building those decoder encoder model in the early days, and we start-- we keep building those model, and we built a lot of them.
- 0:49
Today, we have a family of models, I believe around sixteen we published. We have another twenty coming in the way, and we keep building those models. And you're going to see from this list, those models come in, in two categories.
- 1:04
General model like Palmyra X, Palmyra three, four. We have a Palmyra five's coming soon. And we have a lot of what's called domain-specific model. Creative, financial services, Palmyra Medical.
- 1:19
Now, early twenty twenty-four, basically last year almost, we start seeing this trend with all the LLM basically get very high accuracy in general with any benchmark. We see the accuracy moving and just growing, and I believe everyone noticing this accuracy.
- 1:39
Today, average accuracy for a good general models between eighty, maybe close to ninety.
- 1:48
So that basically make a-- bring a question inside the company saying, "Is it worth it for us to start building and keep building domain-specific models?" If the accuracy today with general model achieving around ninety percent
- 2:05
and we have domain-specific model, should we just keep building general models, fine-tune it, maybe go direction with what you call reasoning or thinking models, and that will be more than enough actually.
- 2:17
Then we don't need those financial or what's called domain-specific model. Now, to answer these questions, we need data.
- 2:27
So whatever we're going to present next actually could be applicable to financial services domain-specific model... Sorry, to medical-specific model, customer support domain-specific model, and all different domain-specific model. Today, I'm going to talk specifically about the financial spec-- uh, what you call it, the financial benchmark for domain-specific model.
- 2:48
Uh, we have something similar for medical, but we believe we are-- but we start seeing similar result. Now, let me dive in.
- 2:59
Just to remind you, we're trying to answer these questions. General model, domain-specific model, should we keep build them? Where are we actually going from here? We start actually saying, "Great, we don't know the answer.
- 3:11
Let's actually do the evaluation. Let's create the data." And we create something called FailSafe. The idea behind it, let's create real-world scenario to evaluate those model, and let's see actually of those new model can really give you the accuracy that we promise or the accuracy that we see today from the benchmarking on domain-specific.
- 3:35
We created two type of categories in this evaluation. Something called query failure. In query failure, basically, we introduce three type of subcategories. Something called misspelling queries. You know, when you go ask the LLM questions, but you do some spelling error, segment error, you do some comma, comment typo issues.
- 3:59
We introduce that to the eval set. We introduce something in, in, like in, in comp-- what called incomplete queries.
- 4:07
You're missing some keyword, some stuff not clear. We introduce what's called out-of-domain queries. If you are not expert in the field, or you decide to copy-paste some general answer, try to answer about something very specific.
- 4:22
And also introduce the second category is what we call the context failure. In the context failure, basically, and this gets very interesting, we introduce three subcategories. What's called basically missing context.
- 4:35
We basically ask the LLM question about context not exist in the q-- the request itself in the prompt. We introduce what's called OCR error. Today, when we do any kind of OCR or convert physical doc-document to text, we introduce a lot of errors like, you know, character i-issues, distance between them, the
- 5:00
word between when you do the OCR could be merged together. So introduce that type of errors. And also we did what's called an irrelevant con-context. Let's say you wanna ask question about specific document, and you end up basically uploading completely wrong document.
- 5:16
Does the LLM going to still answer? Is the LLM just actually figure out you have a completely irrelevant context? Now, when you put all this data together in domain spe-- in financial specific, financial service specific, you need some kind of diversity.
- 5:32
This is a quick screenshot, just tell you what amount of data, how much token. Something worth mentioning, the white paper, the data, the evaluation set, the leaderboard all actually open source today available in GitHub and Hugging Face.
- 5:47
So anyone, please check it out. And we introduce very simple, what you call it, evaluation ma-- key metrics. Basically, we need to look to two things. Either model give the correct answer
- 6:00
Can the model actually give good follow to the grounding or context grounding or basically what we call it here the context? This is quick or high-level way of how we do the calculation.
- 6:15
So to evaluate, we selected a group of models. Today we can see a lot of chart model and also thinking models. This is basically the two lists we have here.
- 6:28
I'm sure you're familiar with this list. And then we run the evaluation and we start seeing very interesting result. I'm going to dive in directly to the result. And basically we start getting something fancy.
- 6:43
With all this color, let me switch to the mo- what basically see.
- 6:49
What started getting very interesting, we're seeing really good behavior in all thinking models. Actually, they don't refuse to answer. This sound good most of the time. But in reality, when you give something those LLMs wrong context, when you give them wrong data, when you have a completely different grounding, those model actually fail, fail to follow this part
- 7:14
and they still give you an answer. And that basically get you way higher hallucination.
- 7:21
If you start focusing on the answer itself, can the model give me answer or not, you can see basically almost every model from the domain-specific to general model, they give you some kind of answer, all of them close to each other.
- 7:34
Actually reasoning or thinking model, they get you even higher score a little bit from there. But we get to the grounding and concept grounding, this is when stuff get more interesting.
- 7:47
You can see specifically in task like text generation or question answering, it's just not performing well. Now all this chart look great. What I prefer is the numbers.
- 8:00
This is the same data we used to generate the chart. We can go through this really quick. And if you look at this number here, especially for example like the o1 or o3 or Palmyra Fin, you can start noticing the stuff those model doing amazingly.
- 8:15
And basically when you ask was it misspelled, when you got stuff incomplete out of, uh, domain, the numbers look amazing. The model can take a query with misspelling, wrong grammars or even out of domain and still can give you the answer.
- 8:31
But when you start going to grounding, this is when the stuff get very interesting. I'm going to hold this slide for a second here. If you-- do you notice something different?
- 8:45
Smaller is better.
- 8:47
Yep. And also those bigger more thinking give you the worst result. You're getting almost seventy-- fi-fifty percent to sixty percent, uh, worse in the grounding. Meaning the model is just not following.
- 9:03
You're attaching context, you ask the questions and the answers exist outside the context completely. Same thing coming to stuff around it, another context. So you can look at the data and see smaller model actually performing better than all this model overthinking at that side.
- 9:22
And this is basically will get us about is this thinking or just a chain of thought? You know, this could be a lot of argument. At least from the data we have in domain-specific task, those model not thinking at that stage.
- 9:39
Meaning hallucination is really high causing a lot of, a lot of issues especially in this fina- if, uh, in this resi- uh, benchmark we run here in fina- uh, financial use cases.
- 9:52
Also we can see there is a huge gap between what you call robustness and the hallucination and getting the answer correct. So definitely we still have a lot work to do to build those model and better performance.
- 10:08
But also that get me to, you know, to the main idea. If you go back real quick here, even with the best model between all the slide, we're still not getting between robustness and context grounding more than eighty-one percent.
- 10:24
Sounds a great number. If you think in reality, you're saying every hundred requests, twenty of them is just completely wrong. So that basically what we start seeing believe at least today with the technology we have, with the current model we have, until we have something completely different, we're seeing you need full stack.
- 10:47
You need the RAG system, you need the,
- 10:50
uh, grounding, you need everything from guardrails and the build around the system itself to actually have something reliable utilized today. In the same time,
- 11:03
I would love to go back and answer the first questions. And our first question here,
- 11:10
do you still need to build the models? At least today from the data we have from running those benchmark, the answer simply yes. We still need to build and continue domain-specific model at least with the today implementation.
- 11:24
Even accuracy is keep growing but the grounding, the context, following all the context correctly, it's still way, way, way behind from everything we see today in the market. [outro music]
- 11:38
Thank you so much guys. [applause] [outro music]