78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble

Picking photos for a dating profile leaves you guessing how they come across. Bumble had experience with what worked in a profile, but members had little guidance from the app when choosing a picture. What could Bumble offer at that moment? Its AI photo feedback feature drew on that experience to suggest how members could improve the pictures they were adding. Each member still chose what went on their profile.

For Supriya Subudhi, a product manager at Bumble, an A/B test offered a way to see whether the advice changed what people did. One group would get the feature and another would use the existing app. At a dating service with over 20 million monthly active users, even a favorable comparison would leave a practical question. Could Bumble keep delivering advice worth acting on at a cost it could sustain?

In surveys and interviews, members kept raising the same problem. They couldn't tell whether their profiles were good enough. Bumble showed people examples of AI guidance, and almost 78% of those surveyed said they wanted feedback. What happened when people could use it? In the A/B test, half got access and half didn't. Bumble reported only 10–15% usage.

The survey measured how appealing the advice sounded once someone had seen an example. Offering help in the app was a different product decision. Bumble chose two moments when members already had a reason to consider their photos.

• While adding a picture. Bumble asked whether the member wanted feedback while they were choosing a photo. They could act on a suggestion as part of the task already at hand.
• When matches weren't coming. Bumble pointed the member back to the feedback on their profile. That gave someone wondering what to change a place to start.

Those invitations increased adoption. Measuring use gave Bumble a way to tell whether offering help at those moments worked.

More people added pictures in the group with feedback, and that group made more profile changes than the control group. But did the advice fit the picture? Some members got a suggestion to smile more when they were already smiling. Counting profile edits told Bumble what members did. Spotting the error required looking at the tip alongside the photo.

The cost of a bad suggestion depends on what someone might do with it. In Subudhi’s Spotify AI DJ example, a listener could dislike two out of four song picks and still enjoy the other two. In her hypothetical banking app, a single suggestion to spend $2,000 the customer can't afford could do real harm. For a dating profile, Bumble drew a line around what a member could reasonably change. Suggestions about skin color or height were out of bounds.

Evaluation tools let the team read the feedback members actually received and judge whether the prompt produced suitable advice. That also changed how Bumble read the aggregate result. Did women and men respond similarly? Would a suggestion written in French come across the same way as one written in English? These were questions to investigate in the outputs and in people's responses before treating an average gain as a good experience across groups.

Bumble started with the large language model it judged most likely to give accurate, high-quality advice. It cost about ten times as much as a simpler approach. Running evaluations on the feedback added a separate expense.

Bumble already paid for machine learning in its matching algorithm. In Subudhi’s comparison, much of that expense came up front in training and compute, with a relatively small additional cost to serve another user. Photo feedback had a different cost pattern. Each request for fresh advice meant paying for another generation. The bill grew both when more people took up the offer and when the same people came back for more advice.

What would justify that expense? Take Subudhi’s example of Amazon’s shopping assistant. If the goal is immediate revenue, extra purchases from good recommendations give the company something to weigh against the cost of generating them. If the aim is to become the store people choose because they can rely on its assistant, the payoff would come over repeated visits. That second rationale might justify a larger budget and a longer wait for a return. A mandate to add AI doesn't tell a team which payoff to look for.

Bumble couldn't afford to scale the first model, so it tried a cheaper one. The bill went down. So did the accuracy of the advice, and the improvement in user behavior shrank. The original A/B test had measured what happened when members received feedback from a particular model. Swapping it out changed what members read, even though the product still offered photo advice.

The evaluation tools now helped Bumble choose what it could afford to run. The team could compare how models performed on different datasets, weigh those results against cost, and quickly retest the feedback after revising a prompt. Bumble settled on the cheapest option it found that still gave a good result. Whether it recovered the full gain from the first model is unclear.

Bumble started in the US and shipped there. Then the rollout stopped. Regulatory requirements differed across markets, and Bumble couldn't take the US feature into other countries as it stood. Subudhi’s advice to product managers and engineers is to consider those restrictions early, while the team can still choose what to build and whom it will serve.

• One version for several markets. Identify the requirements in each intended market up front, then design a version that fits all of them.
• One market first. If the US offers the greatest expected benefit, build the strongest version for US members and accept that the rollout may stop there.
• Different versions by market. Pay the additional cost of making the feature work differently in countries with different requirements.

A positive A/B result still leaves a team deciding which version it can put in whose hands. Subudhi closes with five questions for that decision.