A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo
Read the talk
A Bank Trained Its Own Foundation Model to Catch Fraud
Freddy Lecue explains how Wells Fargo turned transaction tables into a transformer vocabulary, pre-trained a shared backbone from scratch, and used transaction history to improve fraud classification against its production XGBoost baseline.
From a talk by Freddy Lecue
At a glance
Ideas worth remembering
A transaction foundation model can learn from structured financial activity through next-token or next-transaction prediction, then be fine-tuned for separate business tasks.
Tokenization is a central design problem: numerical values, small categories and location information need different representations, and vocabulary size constrains the choices.
The selected model had 1.1 billion parameters. Experiments required substantially more training tokens per parameter than the team’s initial twenty-to-one allowance.
For fraud classification, the model receives a transaction sequence so the new event can be evaluated in historical context. The team credits these relationships with helping detection.
Lecue reports higher fraud detection and fewer false positives than the production XGBoost baseline. Credit-line increases, anti-money laundering and marketing remained under investigation.
One backbone for billions of transactions
Wells Fargo already had tabular data and an XGBoost fraud model in production. The opportunity Freddy Lecue introduces is to learn more from the bank’s transaction history—and make that learning reusable. Debit cards, credit cards, digital payments and checks generate different records across different channels. Separate teams can build separate models around those records, but each then has to solve its own representation problem.
A transaction foundation model would learn a common representation of customer transactions at scale. Teams could then fine-tune the same backbone for credit decisions, fraud detection, anti-money laundering or marketing. Foundation models need not begin with language: here, the raw material is structured financial activity, with billions of transactions arriving each month.
The obvious objection is practical: why replace XGBoost, an established choice for tabular data? Lecue’s answer begins with the existing production system: “We did that.” The transformer approach had to earn its place against that baseline. He previews higher fraud detection, fewer false positives and a latency advantage, then works backward through the data representation and training decisions that made the comparison possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predict the next transaction to learn a reusable representation
The training pattern resembles a language model’s, with a different unit of meaning. A language model learns from language tokens by predicting what comes next. The transaction model instead predicts the next transaction, or the next token within a transaction, using activity from the available channels. That prediction task gives the model a way to learn patterns before it is adapted to a specific business decision.
Next-transaction prediction is an intermediate objective. The useful output is a representation that captures relationships between transactions over time and, as Lecue describes the intent, across customers. Post-training then adapts that representation to questions such as whether to increase a credit line, flag a fraudulent transaction or identify potentially suspicious activity. Predicting a customer’s next action and deciding whether an action is fraudulent are different tasks; the first supplies a foundation for the second.
“From scratch” matters here. Wells Fargo reused transformer architecture rather than an existing model’s learned weights, then pre-trained on its own transaction data using its GPU cluster. The bank was therefore learning the financial representation itself, rather than continuing the training of a language model that already had a different vocabulary and training history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover the raw information beneath engineered features
The input design starts with five questions about a transaction. Together, they describe the event before any particular downstream classifier decides what to do with it:
- Who: The customer or merchant involved in requesting, processing or providing the service.
- What: What is being bought or transacted.
- How: The channel, such as credit, debit or digital.
- When: When the transaction happens.
- Where: Where it happens.
Those dimensions expand into hundreds of features. Existing production models also carried substantial handcrafted feature engineering, which took considerable time to build. The team stepped back from that inventory to identify roughly a hundred core raw features among thousands of available features. The aim was to let transformer blocks learn relationships among the underlying inputs, rather than begin with an extensive set of features already shaped around individual downstream tasks.
This still leaves a representation problem. Even after combining digital, checking, savings and other transaction types, the input has a relatively limited set of characteristics—but their meanings are specific to financial services. Lecue gives that specialized vocabulary as a reason for training the model’s weights from scratch. The next challenge is to express those characteristics in a vocabulary a transformer can process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tokenization carries the difficult design decisions
A transaction table mixes numerical and categorical values. A transaction amount, an elapsed time and a ZIP code cannot all be treated as interchangeable pieces of text. Each needs a decision about what information a token should preserve. Giving every distinct ZIP code its own token is one possibility, but a vocabulary that grows too large makes training harder. Vocabulary design therefore balances the distinctions the model needs against the size of the representation it must learn.
Once that vocabulary exists, an incoming transaction follows a concrete mapping: its characteristics are matched to tokens, those tokens have IDs, and an embedding lookup supplies their model representations. The difficult work precedes the lookup. The lookup applies decisions already made about which values should receive distinct representations and which should share one.
How much vocabulary does each field need? The examples show why the answer varies:
- Small categories: Some categorical fields have only eight types; binary characteristics need fewer distinctions still.
- Numerical and temporal values: Transaction amount and the time difference between transactions carry different kinds of information that need encoding.
- Location and network information: Postal codes and IP-related information introduce other cardinalities. For ZIP codes, Lecue describes regrouping codes using distributions rather than assigning every US ZIP code a separate representation. He refers to a reduced set of 2,800, without specifying the grouping algorithm.
These examples make tokenization part of the model design itself. Grouping location values changes which geographic distinctions remain available to learn; representing time differences makes spacing between events available alongside their other characteristics. Lecue later calls tokenization and post-training the “secret sauce.” The transformer machinery can be familiar while the financial vocabulary requires substantial domain work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A 1.1-billion-parameter model, with more data than first expected
What architecture was sufficient for the transaction task? The team used Chinchilla scaling mathematics as a starting point, then adapted through experimentation in the tabular setting. The selected model had 1.1 billion parameters: Lecue describes it as the smallest model that delivered the highest boost in their experiments. It used a decoder-only architecture for next-token or next-transaction prediction. An 11-billion-parameter version was work in progress at the time of the talk.
The initial data budget used twenty training tokens per parameter. Experiments required substantially more—described as roughly ten times that initial allowance. The useful correction is that a sizing rule borrowed from language-model training did not settle the transaction model’s data needs. The team had to test how much transaction data its chosen model required.
History length was another training decision: how many transactions should fit into the model’s context? That choice determines how much past activity is available when learning a sequence. Evaluation included perplexity-based checks of predictive learning and accuracy dashboards for downstream tasks, keeping the pre-training objective and the eventual business task in view.
The training setup otherwise drew on established large-model techniques. Lecue lists mixed precision, distributed data parallelism and maximizing microbatch size, alongside RoPE, FlashAttention and activation checkpointing. These supported training and memory efficiency; he presents no new training infrastructure algorithm. The distinctive work remained the transaction vocabulary, adapting pre-training to the data and deciding how to post-train for useful tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Judge the new debit-card transaction against its history
Fraud classification supplies the concrete downstream test. The production XGBoost model learned from transactions labeled fraudulent or non-fraudulent. The transaction foundation model brought a sequence representation to the same decision: someone has just used a debit card; given the preceding transactions, should this new transaction be classified as fraud?
The proposed advantage is access to relationships between the current transaction and earlier ones. In Lecue’s account, the XGBoost system needed those relationships to be encoded into features explicitly, a difficult engineering task. Transformer attention lets the current transaction draw on past transactions in the supplied sequence. Lecue identifies this as the team’s explanation for the higher detection rate, rather than presenting an experiment that isolates attention as its cause. The comparison concerns their production baseline; it does not mean tree models cannot use historical features.
At inference time, the input is therefore the transaction sequence, including the new event and its historical context. This makes the history-length decision operationally relevant: the classifier can use the history supplied to it. The model must also fit a card-decision processing window of less than 250 milliseconds. Lecue presents the engineering outcome as satisfactory and earlier claims a latency advantage over XGBoost, but the closing latency wording is internally inconsistent, so it does not establish a reliable measured inference time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Catch more fraud while declining fewer legitimate payments
The result matters along two axes: detection rate and false-positive rate. Increasing detection alone can be an expensive victory if the system also flags more legitimate payments. In the comparison Lecue reports, the transaction foundation model flagged more fraudulent activity while reducing false positives against the production XGBoost model. The account establishes the direction of both changes, without giving numerical rates, evaluation-set details or operating thresholds in the spoken explanation.
How do those two metrics turn into business value? More detected fraud addresses fraudulent activity directly. Fewer false positives avoid another chain of costs: a legitimate customer’s payment is declined, the transaction is lost, and the customer may call the bank, creating operational work. Lecue attributes millions of dollars in savings to the improvement. That is a reported business outcome, without a savings calculation developed in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fraud is the demonstrated use; other teams reuse the foundation
The closing distinction is between building the foundation and adapting it. Wells Fargo pre-trained on billions of its customers’ transactions, using an existing architecture without importing existing learned weights. Lecue describes initial pre-training runs taking a couple of weeks and later runs taking a couple of days. Once the model had learned to predict transaction patterns, that foundation became available for downstream training.
Fraud detection was the completed application presented here. Credit-line increases, anti-money laundering and marketing were under investigation. The same foundation was being used across organizations within Wells Fargo, fulfilling the original goal of sharing learned representations. The fraud result gives that approach a concrete first use; the other applications still need their own task-specific evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:12
I'm going to show you the work we've been doing lastly on building, uh, our first foundation model, uh, a Wells Fargo foundation model. So I explain why we did that, how we did it, and what we accomplished in doing so. So, so first found-foundational model, it doesn't necessarily need to be language-based. I mean, we're all playing with, uh, Claude and, uh, ChatGPT and others. Uh, they're all language-based foundational models. In financial services, uh, we also have language, of
- 0:42
course, uh, from a, uh, structure-- unstructured representation data. But we have also have a lot of structured representation. A lot of this data is tabular data. We have a lot of information from our customers. Uh, we have information in investment banking, consumer banking, and, uh, all of this data is data that we have in, in Wells Fargo and all fun-big financial organizations have too. And, uh, we want to take benefit of all of this huge amount of data that we have. And it's not like you
- 1:12
just take it and put it in a, in a language model and get the best of it. Of course, you can get a lot of benefit out of it, but it's not necessarily the best way to do it. So, so what we decided to do is to look at the data we have, all the transactions, all the customer transactions that they're doing with their debit card, credit card, digital, check, all of those coming from many different channels, and see if we can build a foundation model that we can use and fine-tune for
- 1:42
downstream tasks like credit scoring, or that you can use it for fraud detection, anti-money laundering, do marketing. So how we can build one backbone that you can use and streamline across downstream task. So now you're moving from many models very siloed across different organizations into really one backbone that every organization can just reuse because you learn the representations of customers' transactions at, at scale. So that's the intent. And just to
- 2:12
follow on it, you may say, "Hey, that's tabular data. Why do you just don't use XGBoost and the usual suspect?" We did that. This is the model we had in production and-- But the model I'm going to show you is something that get-- that beats XGBoost, uh, even from a latency perspective, but also from a, uh, from a fraud detections. We get much better false, uh, um, reduction of false positive and better detection rate. So it was a very good lesson learned in term of, uh,
- 2:41
having better model, uh, using foundational model. And to foundational model, I mean transformer blocks, uh, and that, that's, that's what we built. So, uh, so we have, as I mentioned, billion of transactions, uh, on a monthly basis, um, our customers interacting with their debit, credit card, and they do check, and they do different sort of transactions. And the, the question is really how we can use this informations at, at scale and not for one and every single downstream task. So I'm going to cover
- 3:12
a bit of the data, the representations, the architecture that we use, uh, how-- that what the result we get, and the lesson we learned in building this. All right. So, I mean, left-hand side, language foundation model. They're processing language tokens, and then the, the goal is to learn a representation that going to predict the next token. And then you have, uh, you have supervised fine-tuning, and then you do uh, LHF, LRV, uh, and all of the great, uh,
- 3:42
post-training process to get, uh, better at the task that you want to, right? And that's for languages. That's great. Uh, but what if your data is, uh, very much more specialized in term of representation? So that's where we build this transaction foundation model. So instead of having languages as the base, the base is now transactions, transaction from our customers. And, uh, you still try to predict the next thing, and here the next thing is the next transaction or it's the next
- 4:12
token in the transaction, uh, using all of the channel that you have. Uh, but that's not the end goal. I mean, similarly to language model where you try to predict the next token, that's not the end of the story, right? You learn a representation, and then you use that representation, and then you, you, you fine-tune it, and you post-train it for any other task, right? So it could be classification task for languages, but in our context for transaction foundation model, it could be used for, uh, credit line increase, could be used for doing fraud detections and
- 4:41
flagging bad actors from an anti-money laundering perspective. And the goal is to learn this, uh, relationship between transactions over time and transactions across customers. That's really the intent. So the intent number one is you do pre-train from scratch. I'm saying it again because people sometimes confuse. We pre-train transformer blocks, uh, at scale from scratch using our, uh, transaction data. So the weight, the zero weight that we use from open
- 5:11
weights, we just use the architecture and then we, we pre-train it from scratch using our GPU cluster. And then we train it for couple of days and, and weeks. And then, then you do the, the downstream task of, uh, of post-training. But the goal here... And you pre-train it for predicting the next token or the next transactions. And you, you try to, to do this, right? You try to predict the next, the next pa-pattern of the customer. But that's not the intent. The intent is to do the downstream task later on. And in terms
- 5:41
of scale, uh, uh, I mean, we, we operate at a large scale, so we have, uh, billions of transactions, a million of accounts, uh, hundreds of transactions per clients on a monthly basis interacting with their debit card, credit card, or doing check. And, uh, the five main dimensions we look at is really who is doing-- I mean, who is processing and who is asking and who is providing the service. So really the customer or the merchant.
- 6:11
Uh, the, the what, so, I mean, what, what do we, what do we buy? What do we transact? And how, so which channel? Is it credit? Is it debit? Is it digital? When is it happening, and where is it happening? And all of those, they're basically breaking down in hundreds of features. And, uh, if you look at the existing model we have in production, there's a lot of, um, and then people that build, build those model in similar organizations may, uh, may understand that they build a lot of very
- 6:41
handcrafted, handcrafted engineering, uh, feature engineering effort on that. And that's required a lot and a lot of time. So we step back and look at really the raw features we need, not the one that's been probably pre-built and post-built for doing the downstream tasks, but try to understand which raw features. And we end up with, uh, basically hundred features. Even with, at the end of the day, we had thousands of them, but we tried to find which ones are really the, the raw information you need. And then we
- 7:11
let the architecture or the transformer blocks to figure out the relationship between those core features. And, uh, as you can see, uh, we do have a, a, yeah, a bunch of features on f- those five dimensions. And the vocabulary is very unique to the financial services setup. So, and that's the reason why, I mean, we didn't really use, uh, existing pre-trained model, uh, because those-- the vocabulary that we use is very, very specific to
- 7:41
the work that you want to do. And even if you merge, uh, well, all digital and checking and, uh, saving and, um, all of those sort of transactions, even if you merge all of those, you still end up with a very limited number of features and characteristics. So at that scale, then the question is, well, how, how you work with that? So, so we started, everything is tabular data. I mean, there's no secret, it's tabular data. And then the goal is to understand how you're going to
- 8:11
capture the vocabulary, because you do have numerical values, you have categorical values, you have, uh, cat-- I mean, you have information like ZIP code. Do you encode each and every ZIP code in a token? Does it make sense? So there's a lot of efforts you need to push in understanding the vocabulary that you need and the vocabulary that you want also to scale. Because at the training, if you have a too huge vocabulary, it would be kind of messy, uh, to train such, such a model with a vocabulary which is far, far too big. So,
- 8:41
so there's a lot of effort in understanding this, uh, value to token. So which information do you encode in a token? And then you have the token trading, the tokenizations, and then the embedding lookup. So then when we have a transaction coming up, you do have the vocabulary. So it's a-- basically, it's a lookup and aligning, uh, transactions and all the characteristics of that transactions to, to a token and an ID and then an embedding.
- 9:08
So just to give you a number-- well, some numbers. So we have, yeah, quantitative, qualitative, categorical, uh, of different kind, right? So categories with only eight types, some with much bigger types. Uh, you have transaction amount, which is a big deal. You have the delta time between transaction. So you have anything related to IP address, postal code. So just to give you example of, uh, uh, variety and heterogeneity of each and every, uh, characteristic could be encoded in a
- 9:37
binary token or could be enco-encoded in a much larger number of tokens. And even if you look at the b-- the, the ZIP code, I mean, you have a much larger number of ZIP code in the, in, in the US than those, uh, twenty-eight hundred. But you, you have fancier-- well, you have much better way to structure how you're going to, uh, regroup ZIP code based on distributions, for instance. So, uh, architecture. So, so here just, um, what we-- So we use the, the Chinchilla,
- 10:07
um, well, mathematics behind, uh, to define how we build it, right? I mean, initially, 'cause we, we had no clue how it's going to work in a tabular space, even though there are existing, uh, well, well, open weights and open solutions for doing, um, tabular data foundation models. Uh, so we, we, we started from what they learn and adapted to what we needed to do. And that gives you an example of embedding dimensions, number of decoder blocks we used, and, um, all of the,
- 10:38
the recipe from a, from an architecture perspective. Um, so we have-- And a-as you can see, we're not trying to build a trillion parameters model. We don't try to compete with any of the, uh, the, the leaders in the language models because we're working with the-- and focusing on transactions, right? Transactions foundation models at billion level. And 1.1 billion is because that-- the model that give us, uh, that was the, well, kind of the smallest and give us the
- 11:07
highest boost. And now we're working on a, on the next versions at eleven billion. But that's good enough to, to give a lot of, uh, ROI, uh, from a false positive predictions for, uh, downstream tasks like fraud detections and, uh, detection, u-unique detection rate. So, and, uh, and the, the type of architecture, so the decoder-only architecture, uh, that's basically for predicting the next, the next token or the next transactions. And that's really the, the pre-training. So, uh, so
- 11:38
here we initially used twenty times the number of parameters to define the number of tokens we might need. We ended up actually ten times factor, factor than that. So by doing a, a few experimentations, we needed much more, uh, much more tokens per parameters. I mean, this is-- this kind of the initial experimentation we did. Um, uh- So I'll pass on this because this is the, the usual suspect from a, a objective function. So causal language, uh, modeling and math language modeling type of, uh,
- 12:07
um, uh, optimization work. And, and, yeah, from an architecture perspective, nothing really, I mean, fancy here. The, the, the-- again, the most complex part was really the tokenization, and then, and then adapted and doing the pre-training, and then how you do the fine-tuning. So if... I would say the, the recipe or the secret sauce is really how you do the tokenization and then how you're going to do the post, the post-training. But anything from a training perspective, it's, uh, it, it, it-- the question is just how many,
- 12:37
how many transaction you're going to put in your, in your context or in your history for, for doing the training. Uh, and evaluation perplexity-based accuracy, accuracy dashboard on downstream task. That's, that's the usual suspect. And, uh, so yeah. So use mixed precision, distributed data parallel, um, maximize microbatch size, uh, get faster convergence, um, uh, yeah, using Muon, uh, RoPE, uh,
- 13:07
usual and memory efficiency, FlashAttention, activation checkpointing. I mean, here there's nothing really new. It just, uh, we use the, the, the, the, the advices from the, from the best and from the, the one that built even much bigger models. So, uh, I'll pass on this one. Time is short. But, uh, I wanted to spend some time, uh, yeah, on the results. So, um, yeah, so the... W-when you build this foundation, that foundation is
- 13:37
really is, is of, is really, is, is on your backbone. It's something you can use for downstream task. And, uh, the downstream task was to do fraud classification. So you have a transactions, and then you try to predict if that transaction is going to be fraudulent. So in the model we have in production, it's an XGBoost model, and it's trained on large amount of transactions, which is, uh, fraudulent or non-fraudulent. And the missing pieces, that was the, uh, relationship
- 14:07
between whatever transaction you try to predict, try to classify, and the relationship with this transaction with all of the past transactions. And that information is kind of gone when you use an XGBoost model, uh, unless you encode it in a very weird way, which is, which is not so easy. But if you use transformer architectures, you do have these nice, uh, attentions from a, from transactions to past transactions. And that's basically what help, uh, I mean, this is what we think. This is what really help,
- 14:37
uh, pushing the fraud detection rate much higher. But at the end of the day, the, the downstream task was to say, "Here's the sequence of transactions. I have someone just using the debit card. Uh, is that transaction, uh, fraudulent with respect of all historical information?" And, uh, and that's, that's exactly what you put in, in, uh, at inference time when you try to do the prediction. So, uh, so that's the, yeah, architecture. So nothing really fancier. You just need to have the good, uh,
- 15:07
engineering in place to do, um, less than two hundred fifty milliseconds for doing the processes bec- the processing, because the, the, the decline need to happen within that timeframe. But it's, it's, it's... We were, uh, well, ten times slower than that from a latency perspective, so it was all good. And, uh, and just in term of, uh, final result we get. So the main metrics, and those metrics relate right away to, uh, to dollar, to cost, well, to, to savings for
- 15:37
the firm. So it's detection rate and false positive rate. Of course, you can detect much higher, but, uh, in ma-many cases, sometimes you can get much more false positive. And in this case, we managed to get m-- we flag more fraudulent act- activities, and we even reduce the false positive. So that, that was a big win. That was a big win because at the end of the day, if you decline a, a, a transaction which is not fraudulent, uh, then that person is going to call. There is then operation cost associated to it. You lose
- 16:07
the transaction related to the, that non-fraudulent action. So it's a, it's a very, it's a very... Well, something not-- you, you don't really want to happen. So, so again, that's the, that's compared to the state-of-the-art of GV-- XGBoost model, uh, using fraud detection. So it's, it's basically huge. Uh, it's millions of, uh, dollar savings. Uh, so it's a, it's a very big boost. So just to finish up, uh, on that slide. So,
- 16:37
um, so that foundation model has been built on billions of transactions from, uh, Wells Fargo customers. Uh, that's number one. It's, uh, it's pre-trained from scratch, so we don't pre-train from any open-weight existing model. We use existing architecture, but the weight is zero-weight. We pre-training on couple of days. I mean, we did it on a couple of weeks initially, but then that's pre-trained. And then you f- And then when it's pre-trained, it's pretty good in
- 17:07
predicting the next transaction. Then when you're done with that, then you can use it for any downstream task. We did it for fraud detection. Now we investigating for credit line increase, investigating for anti-money laundering, investigating for marketing. So now the same foundation is used across different organizations, uh, w-within Wells Fargo. And, um, yeah. Thank you for having me, and thank
- 17:30
you.