AI Frontiers in Trust and Safety: Combatting Multifaceted Harm on Tinder at Scale

[upbeat music] Uh, hello, AI Engineer World's Fair.

My name is Vibhor, VB for short, and I'm a CI... Senior AI Engineer at Tinder, where I've been working on trust and safety for the last five years. I also work on and maintain some open source AI projects and am advisor to a few AI startups.

Um, but we only have 15 minutes, so I'll jump right into it. Maybe a little bit less than 15 now. Today I'll be talking about AI frontiers in trust and safety, um, combating multifaceted harm on Tinder at scale.

We'll first go over what trust and safety actually is for everyone in the audience, and more specifically what it means at Tinder. Then we'll go over the complex interaction between generative AI and trust and safety.

Some of the problems which most people think about, um, but also some of the tremendous opportunities which will fundamentally change the space. Next, we'll dive specifically into how to actually use LLMs for detecting trust and safety violations in text, covering the end-to-end stack from training to fine-tuning to productionization, and an overview of how we're doing this at

Tinder. Finally, we'll cover what the future looks like for this effort and what we should be most excited about.

So what is trust and safety actually? It's not really something that's well-defined, um, and it's more of an art than a science. Oftentimes, we have to make ethical judgment calls, but it's helpful to look at a breakdown of the goals of T&S by Del Harvey, who led trust and safety at Tinder...

or Twitter for thirteen years. Ultimately, T&S is about preventing risk, reducing risk, detecting harm, and mitigating harm. The ultimate goal is to protect users and also the companies creating the products that they use.

In this presentation, we'll focus on the detecting harm part of it, where we devote a lot of our time to at Tinder. Speaking of which, as the largest dating app in the world, we encounter many, many types of violative behavior on Tinder.

Here are just some of the different categories and a representative synthetic textual example of each. Um, first, we have, uh, social media in your profile, a relatively minor but rather prevalent violation of our pro...

private information policy, um, and it's often done by low-intent users. On the other side of the spectrum, we have things that are low prevalence but high harm, things like hate speech and harassment and pig butchering scams.

So now that we have a sense of what T&S is, let's move on to the problems that generative AI is causing for the industry. One of the biggest problems is that gen AI enables rapid generation of content, which makes it particularly easy to spread misinformation, propaganda, and low-quality spam by drowning out genuine content.

That's what's known as the content pol-pollution phenomenon. Additionally, there's some risk that platforms where the content is posted will essentially inherit the known copyright issues plaguing consumer gen AI tools today, like OpenAI.

Another problem is the accessibility of deepfake technology, which lowers the bar to... of entry to impersonation and catfishing. Um, this also enables malicious interpersonal harm, like in the case of revenge porn.

Lastly, gen AI can be used to scale up organized spam and scam operations. Bad actors can rapidly create profiles by generating text and images, which means that existing signals, the ones we're using today, rely on similarity matching or hashes, um, will be increasingly less likely to work.

As a side note, this is why T&S teams dealing with automated fraud will need to increasingly take advantage of metadata-type signals associated with physical botteneck- bottlenecks in meet space, things like IP address, ISP information, and phone numbers.

Um, additionally, the messages sent by fraudsters on a platform like Tinder can be increasingly automated with LLMs, obviously.

So now that we've covered some of the major problems that gen AI is causing for the trust and safety industry, um, but there's actually some big reasons to be hopeful as well.

So here are some of the opportunities. The first is that AI labs at both startups and some big companies have already done the hard work of pre-training and open sourcing LLMs for everyone's benefit.

Out of the box, these are really powerful in that they have latent semantic capability and often global language coverage as well. Just try doing a few shot example with prompt engineering Llama 3 or Mistral, um, to detect any kind of violation.

It usually works pretty well. By fine-tuning these models, we can actually achieve state-of-the-art performance, um, in some cases better than few-shot GPT-4 performance on downstream textual detection tasks. Um, and the act of fine-tuning, uh, has been made a lot easier because the open source community has produced libraries and tools that are relatively mature and maintained now.

It's easier than ever to do fine-tuning with the low-level details being abstracted away. And there's libraries built for every stage of model development and productionization. Um, the next two opportunities I wanted to bring up are things that every trust and safety organization should be paying attention to.

First is that we're fine-tuning rather than starting from scratch. And because of that, um, and the strong open source library support, we can actually develop... accelerate the model development process from months to weeks.

And, and additionally, one of the major reasons we see such good performance from the fine-tuned open source LLMs is that in general, model performance degrades quickly in trust and safety due to the adversarial nature of automated fraud.

For example, whenever we release a rule to block one spam wave, bad actors are incentivized to and ultimately do change their behavior to get around it. Um, but the generalization performance of LLMs slows the model degradation curve significantly, and we basically get this for free when we use LLMs.

Okay. So let's move on to some of the specifics of actually using LLMs for T&S violation detection. The first major step is creating our training dataset. This is often the hardest part.

Um, that's due in part to how easy some of the later steps are, as we'll see. Um, but it's also because, uh, smaller datasets are required for fine-tuning versus training from scratch.

Um, in some cases, hundreds to thousands of examples. And this necess-necessitates creating a pretty high-quality dataset. Um-

What does this dataset look like? Well, GPT-type LLMs, like the closed source GPT-4 and Claude Opus, um, and also like the open source Llama Mistral models can be thought of as a text in to text out.

This is an approximate mental model, but it helps for understanding what the dataset should look like. In our case, the text in is the potentially violating text we wanna make a-- we want the model to be able to make a prediction on, wrapped by some prompt.

Uh, and the text out is a classification label or, um, some extracted characters representing the violation. It's not a pretty-- not a very complicated format, um, and there's an example for how our-- a synthetic example for how we would, uh, detect users if they're underage in their written bio.

Um, as for actually assembling this dataset, it's possible to do it entirely by hand because, again, we only need hundreds to thousands of examples. Um, one thing we've worked-- seen work quite well is actually to incorporate the largest LLMs in the data generation process.

We could generate purely synthetic training examples with few-shot prompting. Um, this introduces some risk that the data won't resemble the true data distribution, um, and it's platform-agnostic. What we found works better is to actually do a hybrid process where we can use GPT-4, um, with some clever prompting to ensure we don't run into the alignment, um, built

in, to actually make predictions on internal analytics data and to mine examples for our training set that way. The cost of doing this is inversely proportional to the true prevalence of the harm, um, but that cost is still pretty negligible, um, and it provides a metric that alone is actually really helpful to track for TNS operations teams

anyways. It's possible to use heuristics to restrict the LLM calls to more likely candidates as well. Um, and finally, when we get the mined examples from, uh, using GPT-4 effectively as an auto moderation, uh, we can then do a manual verification, fixing any mislabeled data and making judgment calls where the label is more ambiguous.

Okay, so got the training set. Now let's talk about fine-tuning. One question you might have is, why don't we just directly use the API LLMs like GPT-4 to do this detection in production?

One fundamental reason is scale. Tinder has a huge real-time volume of profile interactions, and hitting OpenAI APIs that often doesn't scale in terms of cost, latency, and throughput. The other reason is maintainability.

By fine-tuning our own models, we have full control over the model weights and can re-fine-tune when production performance inevitably degrades, um, without needing to worry about changes in the underlying base model.

One additional benefit for, for us for classification tasks is that we have access to the output probability distribution, which means, uh, we can use the confidence-- We can create essentially a confidence of the prediction, like in a traditional machine learning model.

As for actually doing the fine-tuning, well, the relatively mature open source ecosystem makes this really easy. Hugging Face libraries make this as simple as writing a few hundred lines of code without really needing to understand anything about deep learning or transformers.

Um, and we've also had particular success building out training pipelines in notebooks. There are also libraries which abstract fine-tuning to just config files like Axolotl, Ludwig, LlamaFactory. Um, and finally, there's managed solutions emerging that provide additional UI and dataset management support, um, for rapid experimentation like H2O LLM Studio and Predibase, uh, the latter of-- with whom we've

had a lot of success with. Many of you are probably also familiar with parameter-efficient fine-tuning. Um, this is critical for us. Uh, low-rank adaptation or LoRA freezes the weights of the base model and can create a fine-tuned model while only really needing to learn a f- megabytes of additional weights.

Accordingly, the fine-tuning can be done quickly, um, and only on one or a few GPUs, which enables rapid experimentation and also unlocks using larger base models. Um, PEFTs of larger base models are more likely to be better than full fine-tunes of smaller base models, especially for classification tasks.

We've had a lot of success also with QLoRA, which unlocks fine-tuning on a single GPU, um, even the largest models. Lastly, one of the biggest reasons to use LoRA is that we can take advantage of the massive inference optimizations, as we'll see now.

Um, so production. Um, in production, we use LoRAX, which is an open source framework that allows users to efficiently serve thousands of fine-tuned models on a single GPU. It exploits the fact that a fine-tuned LoRA adapter, which is just a single fine-tuned model, is only a few megabytes in size.

Many adapters can be efficiently served jointly by simply shuffling and batching adapters and requests for efficient serving. In practice, this means that the marginal cost of serving a new adapter on the same base model is virtually zero.

I just want to let the implication of that sink in. It means that we can train adapters for the many, many different types of trust and safety violations possible, hate speech, promotion, catfishing, pig butchering scams, and so on.

And we can serve all those adapters on a s- on one or even a small set of GPUs and not need to worry about horizontal scaling. Incorporating a new adapter in production is as simple as storing the megabytes of weights on some file system and modifying a request to the LoRAX client.

Special thanks to the Predibase team who developed and maintains LoRAX and have provided us a lot of support in it.

The, um, optimizations in LoRAX basically enable us to do real-time inference, and at Tinder, that's critical. We can support on seven billion parameter models, uh, tens of QPS on a hundred-ish milliseconds of latency on A10 GPUs.

This is good enough for some use cases. Um, and for those other use cases, the high-frequency domains, we can, um, further reduce throughput by gating requests with heuristics. Um, for example, uh, for detecting social media in profiles, um, we can make predictions only on bios that contain some word that's not in a dictionary.

Um, and then we're also exploring doing cascade classification through some distillation process where we train adapters on smaller base models optimizing for recall and only train-- only call the larger base model adapters when the smaller one gives a high enough score.

Another advantage for us in this T&S space is, in general, LLM outputs are computationally expensive because the generation is done autoregressively one token at a time. But classification or extraction tasks require only, uh, exactly one token or a few tokens to output, which means our time to prediction is low.

Um, and compared to NLP models of the past, we're seeing that we can get massive improvements in precision and recall just due to the much higher latent semantic capability of today's LLMs.

We can achieve near 100% recall in simpler tasks like social handle detection and significant improvements over the baseline in more semantically t- complex tasks. The other huge benefit that we get is way better generalization performance, which I've talked about a bit before.

In particular, this is important for T&S because, uh, it's, it's an adversarial game. Bad actors and other violative users always try to avoid detection, for example, with intentional typos, mixing letters and numbers, and innuendos.

But LLMs are much better at making sense of these, meaning that these new adapter-based models get stale less quickly than other traditional machine learning models and are a better defense against harm in the long run.

So, uh, where do we go from here? We're interested in the growing work on non-textual modalities and how we can leverage that for detection purposes. For example, we can use pre-trained visual language models like LLaVA to do explicit image detection, and that's an active area of exploration for us.

Overall, we're excited about rapidly training adapters for detecting harm along the long tail of T&S violations. Uh, we can create high-quality datasets with trust and safety operations and policy experts, um, with that AI in the loop.

We can automate training and retraining pipelines for fine-tuning adapters, and we can take advantage of LoRAX to slot in new adapters for inference with low marginal cost. Ultimately, we can build a next-generation defensive moat against harm that takes advantage of the gen AI landscape today, ultimately leading to a safer, healthier platform.

Thanks for listening. [upbeat music]