Navigating Challenges and Technical Debt in LLMs Deployment

[on-hold music] Hi, everyone. I'll try to touch on three main things and, uh, mainly how AI moved from excellence in structured data to LLMs and the use of unstructured data

that most of our organizations have. And also we'll touch on intelligence augmentation and really the hype around AGI and, and the doomsday. Uh, and finally, we'll, we'll, we'll briefly talk about the challenges and technical debt and, and highlight the findings, um, that we have, uh, published recently.

So over the last ten to fifteen years, uh, most of the AI values that we have seen is really coming from structured data. And, and we have seen, uh, supervised learning and deep learning doing really well at labeling things.

And, um... But this is not the reality. Like, the... Most of the organizations, it, it's estimated that most of the organizations' data is unstructured. Specifically, more than eighty percent of the organizations' data is unstructured.

Uh, and it's also estimated that seventy-one percent of them really struggle in managing and, and securing this kind of data. Um, and it would have been ideal to really build automated systems, try to do certain recommendations, uh, based on this data.

Uh, but now it's easy to, to really use it and, and have it to conte-contextualize, uh, or customize the contextual language models. Uh, so you can easily have this as an extended memory to, to your language model, uh, and have it formulate answers based on the domain-specific data, uh, that you have within your or-organization.

Um, and, uh, talking about the AGI and, and, and the, the way we see, uh, uh, uh, LLMs or generative AI in general at Mastercard, it's really augmenting human productivity.

Uh, and, and we have seen a lot of hype around, you know, uh, generative AI is gonna replace our jobs and, and doomsday, and it's taken over. And I, I recommend you this great article from, from Nature, which is really talking about, uh, stop talking about tomorrow's AI doomsday when AI poses risks today.

So stop doing speculations about what AI will become tomorrow and what kind of risks, uh, that, that it will have tomorrow, and really focus about the, the current risks that it poses today.

And, and funny enough, some of the, the, the big speakers about the doomsday are actually ones who have AI systems out there to the end users with, with a lot of risks, as we have seen, uh, in the past.

Uh, and, and this, of course, will help regulators as well be more focused. Like, if we highlight the, the current risks, and so it will help them more focus to have the laws and the policies that can really help them regulate the current AI systems and at the same time be, uh, early sort of, uh...

or proactive enough to, to adopt any new laws whenever new algorithmic approach, uh, come up. Uh, also when it comes to the algorithmic foundation, like, you know, like, uh, you know, AI and generative AI specifically has been transforming our lives in so many ways, but the, the algorithmic foundations itself behind LLMs is not really the ones that

will get us to AGI. And also I, I recommend this talk from LeCun, one of the fathers of machine learning, where he talks about the objective-driven, uh, learning, um, and the whole idea that, you know, you know, despite the fact that it's transforming our life in so many ways, it's really so dumb at the core of it.

Uh, and it's... Because of the whole idea that it's autoregressive, and whenever it's making a mistake, this mistake really amplifies over time because the other generation of tokens is so dependent on what it's already generated.

And I can't help but by, by, uh... but quote, you know, bring this quote from, uh, Ada, uh, Lovelace, uh, otherwise known as the world's first computer programmer. So in, in her 1843 Analytics Engine paper, she mentions that the analytical engine or machine learning, as we call it today, cannot originate anything by itself.

Uh, it can only do what, what we ask it or what we order it to perform. Uh, because basically we don't have this algorithmic foundation that can really get us to something that can originate something by itself.

And despite being, you know, about a hundred and eighty years old, this statement still holds, uh, despite the transformations that we have in so many, uh, AI algorithms and applications.

And, um, funny enough, like, I've met a lot of people that thinks OpenAI is the one behind language models, and I, I do hope that you folks don't share the mi- the same misconception.

Uh, the whole idea of predicting the next token given a specific context is very intuitive and simple idea, uh, that it's not only a few years old, it's few decades old.

Uh, but was... What was really broken with this is, is the whole user interface. Um, and, and a lot of folks have really, uh, misunderstood what ChatGPT is all about.

So ChatGPT really fixes this whole user interface idea, that you are able to naturally, as, as we speak, be able to prompt the, the LLM in, in a natural way and, and get your response.

And this is what was, was really broken with the, with the language models before GPT assistants and the ChatGPT specifically. Because this kind of data is really rare and, and, you know, LLMs or, or, uh, OpenAI specifically have built their base model based on the internet scale data.

But then in subsequent phases before they released this GPT assistant, they had to go through, uh, outsourcing a lot of, a lot of folks to really go about generating manual pairs of responses and, uh, questions and responses.

Uh, and, and as I said, like, you know, LLMs despite, you know, being dumb at the core of it, it's really accelerating, uh, innovations everywhere. And, and we have seen great adoption in, in so many industries.

And Mastercard is no different. Uh, so we have been de-risking this technology responsibly, of course. Uh, and we have a recent press release, uh, in Feb, uh, where our president, uh, announced how we used LLMs, generative AI specifically, to boost fraud detection, in some cases by three hundred percent.

And to go into the, the, the last topic of my, uh, of my session is, is basically about the challenges. So let's first, you know, understand the essentials that anyone needs for building, uh, a successful GenAI application.

So basically, you need to have access to a variety of foundation models, and you need to have an environment to customize contextual LLMs, and you need to have an easy-to-use tool to build and deploy applications.

So basically, all the, you know, the, the widely used tools that we have seen before GenAI wasn't really applicable to the GenAI landscape. And finally, we need to have a scalable ML infrastructure that can really help in scaling up and down, not just creating replicas, but really creating replicas at a speed that can work for our, uh,

for our end users. And I've tried to color code the different essentials based on the challenges that we, we would see in, in, in building such applications. So access to a variety of foundation models is, is, is not so challenging.

Yes, still you need to do this kind of trade-off between cost and the model size, but it is available. And the environment to customize the, the language model itself is, is, is a bit challenging because, yes, we-- most of the enterprises have their own AI environment, but it is not really something that is built for, uh, such

models, such large models. And, and the easy-to-use tool, I think is the most challenging part of the whole equation, because n-none of the tools that we have seen before, and, and most of the tools that, that most of you guys use now, is really, is really as new as LLMs.

None of them has existed before. Uh, and finally, the need to have the scalable ML infrastructure is, is a bit of a challenge as well. Uh, and, and we have seen this nice curve from OpenAI, where this shows that the GPU compute and RAM, uh, for inference is actually getting more, uh, or greater than the, the, the

compute they use for training the model itself. Um, and, and before I, I, I talk about the, uh, the challenges in LLM and highlight the papers that we have recently published, uh, I just want to bring up this really nice chart from the NEPS paper, two thousand and fifteen paper.

And it shows that ML code, which is at the core of building any machine learning system, is only a small fraction of what goes into building the end-to-end pipeline.

And specifically, it's less than five percent of what goes into building the end-to-end pipeline. And this is what I call-- like, I've met a lot of folks, uh, during, you know, before my talk, and, and they think that, you know, an AI engineer is all about really, you know, connecting APIs and, and getting this kind of plumbing,

uh, in place. But I think it's more than that. It's, it's really everything around this ML code box. It's really building this end-to-end pipeline, which is-- accounts for more than ninety-five percent of the work.

Um, sorry. Uh, so before the challenges, so we'll just highlight the, the two different approaches that are widely used by, uh, in, in the industry. So the, the first one is, is really the closed book approach.

So you have a foundation model, you use it as it is, zero-shot or few-shot learning, or even fine-tune it with your domain-specific data. And, you know, if you ask any of the, the folks in the enterprises, they will tell you, "We really have hard time operationalizing such models because we have certain accuracy constraints."

So basically, the hallucination, and they do it very, you know, confidently. Uh, attribution. Um, uh, you know, we can't really understand why the models are saying what they are saying.

Uh, staleness. They go, they go out of date, and we have seen, uh, the different releases that, uh, that, uh, comes out of OpenAI. Revision. As, as you know, in GDPR or even in, in California AI law, uh, folks can opt out of the AI systems, and, and their information can't be used again for training or, or

influencing the model decisions. So you need to be able to do the model editing. And, and this is really hard in, in the foundation model or even if you fine-tune your model.

Um, and finally, customization. So you need to be able to customize these models with your own do-domain-specific data and have it really more grounded or more factual to generate information only based on your info-- your domain-specific data.

And it turned out that the, the solution to all of these problems is really to couple the foundation model to an external memory, uh, also known as the RAG.

Uh, so RAG, as you can see, that, you know, the, the original setup remains as it is, but we have added this additional context, which is coming from your domain-specific data.

Um, and it, it is grounding, so it's improved the, the, the factual recall. Uh, there is very nice paper, uh, around, uh, RAG augmentation reduces hallucination in, in conversation.

It, it kind of rhymes, but, like, it's very nice and shows how this kind of architectures really reduces the hallucination of the LLM systems. And you can also have it up to date.

So you can easily swap in, out ver-vector indices. So you can do the revision, you can do, uh, attribution, of course. Like all of the problems we have mentioned in the previous slide, you can also do as part of this, uh, RAG setup.

So y-you have access to the sources coming out of your retriever, so you can easily go back and understand why the model generated certain, certain text or certain, uh, decisions.

But i-i-it's not so easy, right? So, like, there are so many questions that need to be answered for this system really to be optimized and, and be able to work in production.

And this is not even half of the questions that, that we have out there. So mes- mostly, how do we optimize the retriever and generator to work together? Uh, so despite, like, the mainstream, uh, kind of RAGs that z- z- most of the people are doing right now is really having the retriever and, and the generator as

two separate brains that don't-- that none of them knows that each other exist. Uh, but the actual RAG paper, uh, that was released, uh, by, by FAIR is, is actually about training these two in, in parallel.

So you need to have access to the model parameters, and this is now thank- thanks to the people who are believing in the open source, uh, is possible. Uh, so you can have access to the model parameters, the open source model parameters.

So you can fine-tune the generator to generate factual information based on what it gets from the retriever. So it's not just, you know, attaching an external memory and, and, you know, two sides of the brain that, that, that are totally separated.

So this is our paper. Uh, so it's, it's very similar to, um, the, the NIPS one, but it, it, it really shows the unique and different challenges that, that we would see in, in building an end-to-end LLM application.

So you can see that, you know, the, again, the, the, the surrounding boxes around the LLM code or the adoption of foundation model is, is really, you know, accounts for more than ninety percent of what goes into building such application.

And it's not really just about, you know, if, if we pick one box about the domain-specific data collection, it's not just about building, uh, or generating the domain-specific data.

It's also how do we, how do we preserve the access controls within our enterprises into the, uh, into these ecosystems. So, like, you know, I'm sure most of the organizations that you work with have access controls.

Like, you can have access to certain systems but not others. So how do we make sure that we don't have a global LLM system that can really have access to all of the data that we have behind the scene?

So we need to maintain the same access controls, uh, and, and have certain specialized models that can work for certain tasks. And also, you know, coming back to this Nature article that we need to focus about the, the current risks that AI poses today and how we build safeguards around it.

And this was really the core, uh, I, I would say, you know, principle behind Mastercard to move to, to adopt LLMs. So we have the seven, the seven core principle of building responsible AI.

And, and it, it, you know, it's all everything around privacy. It's around security, reliability. Um, and, and, you know, we, we, we also have this governing body and clear strategy that really enforces these core principles into, into the building of such LLM applications.

So yes, we can go about really de-risking new technologies such as LLMs and use it for some of the services that we have. Uh, but at the same time, we need to have the right safeguards to really make sure that, you know, the access controls are in place and also we are not, you know, generating any biased

information. And, um, so funny enough, one of the reviewers, one of the reviewers who accepted this paper, uh, mentioned that, you know, after he, he read the paper, he was wondering, uh, if, uh, if LLM is, is the right tool to use for, for solving some of the applications given the huge number of challenges and technical debt,

uh, that, that, uh, that he have seen. Uh, but as the saying goes, like, you can't make an omelet without really breaking a few eggs. You can't really use this kind of transformative technology, uh, in your business without really being challenged in so many ways.

Uh, and that's all I have for you. And do check out some of the boxes that we have from the AI engineering team, uh, from Mastercard. It's all about putting AI in production and, and the whole other boxes around ML code or the LLM, uh, that we have seen in the figures.

Thank you. [outro music]