How to build an AI-Native Health Company

Talk transcript

1. 0:00

   [upbeat music] It's time. We can get it started. Uh, I'm Dan.

2. 0:15

   I'm from Maven Clinic. Today, we'll share the experience how it's transitioning from a traditional technology company to AI-native company. Before I start it, I would like to do a little exercise.

3. 0:28

   Raise your hand if you think you are already AI-native company. [laughs]

4. 0:32

   Okay, we saw a few. Raise your hand if you thought about it but haven't started the journey yet.

5. 0:40

   Okay, we saw a few. That means most of us is in between. Hopefully, this talk can help you with that one.

6. 0:47

   Um, so Maven Clinic is a large digital health platform. We are focused on [REDACTED:gender] and their families. So we specialize in, like, maternity, fertility, parenting, menopause. We start our AI journey just two years back, and this moment we build something called Maven Intelligence.

7. 1:08

   It's our orchestration layer across all our product to enable AI for everybody in this company and for our clients.

8. 1:17

   So AI is here and improving every day. I think adopting it is not optional. Even you choose not to, your competitors will do. This is a quote I heard, like, a couple years back I would like to share here again.

9. 1:32

   Like, "Tractors won't replace farmers, but the farmers who can operate the tractor will replace the ones who cannot." Hopefully, everybody here will become farmers who can operate your tractors.

10. 1:44

    That's the goal. So first of all, I don't think there's one single definition what it means by AI-native, and more importantly, there's no predefined playbook you can just follow, and bingo, you become AI-native.

11. 1:59

    For us, it really come down to three parts. One is internally, we adopt AI tools whenever it's possible. It can be as simple as, like, generating your daily summary, managing your meeting, create a Jira task.

12. 2:14

    Anything you need to do today manually, you should think about to say, "Can I use AI to do it?" Whenever you want to ask other people to do something for you, you should say, "Well, I can use AI to do it."

13. 2:26

    A lot of leaders, in fact today at Maven, include our CIOs, CEO, they use AI tool to solve those tasks by themself now instead delegate to other people. That's for internally.

14. 2:38

    Externally, we want to build AIs into our product.

15. 2:42

    We achieve two goals there. One is really focused on improve our user experience. Second, maybe help us reduce our operational cost. Like, uh, like, AI-based, like, chatbot is really good example.

16. 2:56

    It's twenty-four/seven always available, can help our address issues, help our customer instantly. It's way better and cheaper compared to human agents. Certainly, and I think it's more important, we need to think about culture, process, the way we work, how we can change it so we can maximize what AI offers for us.

17. 3:18

    I will touch it more on the following slides.

18. 3:22

    So when we come to adopting new technologies, there's always, like, group-- three groups of users. One, there's some early adopters, right? For them, we don't need to do too much.

19. 3:34

    Only thing we need to do is enable the tools for them and encourage them to share what they learn with the company. And what we need to really focus on is the one in the middle.

20. 3:46

    That's the majority. We should build a shared, uh, infrastructure for them, build easy-to-use tools for them. Just make the adoption as seamlessly as possible. More importantly, we should really listen to them, get feedback, consistently improving.

21. 4:03

    For example, last year, most of the folks at Maven they are using Cursor. This year, a lot of them switched to Claude Code. For us, we need to support both.

22. 4:13

    We need to meet where they are. Just make sure they're comfortable to use it. And for-- In all the places, you always have a few slow adopters. They always have concerns, worries for the new technologies.

23. 4:26

    For them, we just should meet where they are, understand what their concern is. But more important, we should be crystal clear with them where the company is heading to.

24. 4:39

    So AI is really good at execution if we know what we want to do. So this change how we should hire new people and how should we reward it.

25. 4:50

    We used to-- The way we used to work is we have a senior engineer assess the problem, come up with solution, and delegate to other engineers for implementation, so we can, uh, work on in parallel and be faster.

26. 5:05

    But these days, we found less and less engineers who would like to dedicate implementation work to other people because these-- they already figured out how to solve the problem.

27. 5:16

    They just use AI to solve it instantly. Dedicating to other people means more overheads and way less efficient. That also means, like, when you have a new people, you want to make sure they can solve the problem independently.

28. 5:31

    They pretty much has to work as a traditional technique lead member. We can't-- We cannot afford other people dedicate an implementation task for them. Also, when we hire new people, we should think about what we are looking for.

29. 5:44

    We definitely want to look for somebody genuinely interested in AI. The domain is moving so fast. We want them keep learning, also help the team to stay on track.

30. 5:55

    Secondly is with AI, engineers can do way more than they used to do. The boundaries between PM and engineers is getting blur, blurry. We found, like, engineers who really understand the product.

31. 6:10

    In fact, they can have more-- way more contribution than the traditional engineer who only focus on software side, and this is what we are looking for.

32. 6:19

    And those deep understanding of the system, the ability to can handle complicated, ambiguous problem is also very valuable. This is where AI 9-call. When we hire new people, this is also the people we are interested in bring on board.

33. 6:35

    For people we bring them in, we want to re-reward them in the proper way. Even in our performance res-review, we start to ask, "Okay, what you have done for AI side?"

34. 6:45

    We definitely want to reward the people who leverage AI to multiple their impact, although this is impact for everybody in the company.

35. 6:56

    So now we have the right tools, we get the right talent in the place, and we need to change how we work to maximize the benefit of AI. The-- we used-- The way we used to work is to say, okay, we spend the weeks, sometimes even months, to flesh out the business requirements, finalize the design, and then

36. 7:18

    do the implementation, because implementation can be really expensive. If we didn't get the other part right in the beginning, it's, it can be very costly to change it later.

37. 7:30

    But, in fact, we never get the things done right at the beginning anyway for any our big project. With AI, building is super fast. It's probably couple minutes you can get it done.

38. 7:43

    Argument is really expensive one. So we should s-really think about what's the best we can work, how can we deliver fast. It's still okay, you can think about what you want to deliver in one year.

39. 7:56

    You can assume AI models can do anything you want in one year. Based on that one, really dream big to think what you can do in one year. But it should only serve as inspiring, inspiring and directional.

40. 8:11

    What we really need to focus on is what we want to deliver in the next two to four weeks, right? What we want get the PMs and designer is say, "Okay, tell me what I need to d-deliver in this sprint."

41. 8:23

    And the engineer will focus on it and get it released if in end of sprint, if not sooner. Meanwhile, and the, the PMs, they have time to flesh out the next batch of the requirements.

42. 8:35

    If and under the sprint, they say, "No, what we decided two weeks ago is wrong," it's totally okay. We can re switch the gear, get it fixed quickly. That also means, like, uh, we prefer people not to write pages or pages of PRD or TDD anymore.

43. 8:52

    We page-- prefer them to write just a short one or two pages. Everyone is really serve as communication, so we can iterate on it. The really awkward part is the midterm goals.

44. 9:05

    Those like three months, six months. It's very hard to plan these days. The reason is, I don't know what AI models will be capable in three months. There may be multiple releases already.

45. 9:16

    So we prefer not to focus on this one. But this one can maybe make it not very easy for most of the folks wh-who has been in this, uh, domain for a long time, because traditionally, we get used to have a quarterly planning, or we plan it for six months.

46. 9:33

    But it's our job to get u-used to the new AI era and learn how to work it efficiently.

47. 9:42

    So I want to talk about coding and software development a little bit more here. AI coding tools is probably the most successful AI application, and it's really good at implementation.

48. 9:56

    So you probably heard a lot of people say, "Okay, I have th-these AI tools. Now I can u-even use my phone to implement software and automate every stage." If they feel comfortable to do that, it's totally okay.

49. 10:09

    But you don't have to, what I'm trying to say here is. At Maven, this is our journey, how we adopt those AI tools. We started with the lowest risk task, like starting with writing unit test documentation.

50. 10:23

    Those things are very easy to verify, and the risk is super low. By doing that one, we build the confidence, and we start to construct our own rules, skills, and build our guardrails.

51. 10:35

    And then we push to the whole engineer team say, "Now you should use this AI coding tools for all the tasks." When they choose not to, is the time we really want to learn, say, "Why you don't do it?"

52. 10:47

    And the-- and this moment, we pretty much use the AI coding tools to do all our implementation. Engineers really focus on reviewing, architecturing, and evaluation.

53. 11:00

    So, and with AI coding tools, we are writing so much code these days. Code review becomes really challenging. So for a good engineer, used to they probably write hundreds of lines code every day.

54. 11:13

    These days, they can easily write, like, thousands. If we capable do the code review as we used to do, we won't be able to keep up. We also try the multiple, like, AI coding review, review tools.

55. 11:27

    It helps a little bit, but we don't feel comfortable one hundred percent rely on them yet. We still found the feedbacks from our engineers are very, very valuable. That means we need to really change the way we are doing code review to meet where we are now.

56. 11:43

    And the couple things we have done. One is we allow engineers to self-identify whether they still need code review. If they think, uh, this PR is simple enough, I feel very confident, I don't need anybody take a look, and we're fine with that one.

57. 11:58

    We let them merge, but we still hold them accountable. And, uh, if they do want code review, we want them stay with the best practices. For example, each PR shouldn't have more than five hundred lines of code because nobody can do a meaningful code review with the ones has, like, thousands of lines code.

58. 12:17

    And we also enable the, like, uh, stack the PR What it means is that for big feature and the engineers can bring-- break it into multiple PRs while people review the PRs, and they can keep working on it.

59. 12:31

    One thing we really want to avoid is the rubber stamp we call it. Means like people submit the code review, you cannot really do anything to it, you just say blindly approve it.

60. 12:42

    This is the worst case we should really avoid because that's just give us false confidence. We think we review it, it's good, and we release it. Meanwhile, we should keep working on our AI code and review tools because we are think that's the future.

61. 12:58

    So at this moment, we use AI tools pretty much assist on each step of our software developments. Our goal is it will be automate the, the whole life cycle from end to end, from designing, implementation, until it's fully released.

62. 13:15

    More important, we want the AI tools to be able to monitor the live traffic and be able to catch the issue early and automatically fix it. That's what we are still working on, and we are not there yet.

63. 13:29

    The last thing I want to touch a little bit for this presentation is about reliability. So what it means is like, for the traditional software, it does what we implement there.

64. 13:41

    No more, no less. But for the GenX solutions, ha-ha-hallucination is there. We cannot ignore it. And there completely eliminating them is can be very costly. Sometimes it's not necessary either.

65. 13:57

    So the way we should do is really have a holistic solution, even from the ge- beginning. For example, we can start with identify which failures is acceptable, which ones are not acceptable.

66. 14:11

    For our AI system, for example, we have the functionality to help our customers to schedule appointments. If we fail one out of one thousand, probably it's okay. I'm not saying it's a good experience, but users usually can just click the button again, we will reschedule for them.

67. 14:28

    Probably it's okay. But if we help user to like, uh, submit their reimbursement claim, we cannot tolerate a failure. Because if people ask of two hundred dollars, we issue them fifty, or they ask fifty, we give them two hundred.

68. 14:43

    Each case will cause a escalation right away. For those cases, we have to put in extra steps. For example, when we receive their receipt, we will use different models to review the same receipt.

69. 14:56

    We only move forward if the results from different models agree with each other. If we really have trouble to figure it out which one is right, it's, it's easy-- it's okay to tell the customer, say, "Hey, we trouble to process your stuff.

70. 15:10

    Do you want us to get you connect to a human agent?" We will move from there. That's acceptable solutions. And also, we ha- should have a rigorous process to release our software.

71. 15:22

    For us, we have like hundreds of integration tests for which pretty much covered all the use cases we know, and we are keep adding to the integration test suites.

72. 15:32

    And the-- when we run the integration test, not only pass once is not good enough anymore because the LLM can do different things. So for each test case, we run it many times.

73. 15:44

    We consistently requires a high pass rate, like for example, ninety percent for all the time.

74. 15:50

    And more important, and after we launch the software, we have our auto-evolve system evaluate-- carefully evaluate each conversation. We have predefined a lot of rubrics, what we think is good, what is bad.

75. 16:06

    And then we will generate results. We will re-review the score. Besides this one, we also have a dedicated, uh, a group. Their job is manually review those conversations. We will spot check our conversations.

76. 16:21

    That helps us to say whether we need to come back to improve our systems or our rubrics is too strict or too loose, and we need to consistently improve it.

77. 16:31

    When we launch new features, that's the time we say not on-- spot check probably not enough. We really want to re-review like, say, twenty percent, and we can do it.

78. 16:41

    This whole process [clears throat] makes sure we feel really confident we never release something although we know hallucination is there.

79. 16:50

    That's pretty much I have for today, and I can stay here to take up questions, and if you have other things, you can reach out to me. [audience applauding] [upbeat music]