The Multimodal Future of Education

Talk transcript

1. 0:00

   [upbeat music] Uh, hi everyone, I'm Stef.

2. 0:16

   I'm gonna talk about the future of education with multimodal AI. Uh, we are here at AI Engineering Summit, and AI engineering starts very early. So I'm curious, how many of you have kids?

3. 0:29

   How many people in the room have kids?

4. 0:32

   Okay. Wonderful. Uh, how many of your kids, uh, have played with generative AI so far?

5. 0:40

   Okay. So, um, you won't be surprised [REDACTED:age] see the next slide. Basically, seventy percent of generative AI users, uh, are from Generation Z. This is a recent study from Salesforce.

6. 0:52

   So it starts very early, and the reason I care about the future of education with generative AI is because edu- education needs a wake-up call. So we know that early litra- literacy rates, um, need [REDACTED:age] be improved around the world.

7. 1:07

   Only seventy percent of [REDACTED:age] actually can read and understand a simple story. At the same time, we've seen a big gap in learning that happened during COVID. Sixty percent of children and teenagers are left behind, in particular in math and reading.

8. 1:23

   And it's not just K through twelve. For older, uh, people, for adults, like, we need [REDACTED:age] do a lot of reskilling. So

9. 1:33

   multimodal AI has a potential [REDACTED:age] transform education. Uh, and we are-- we know that students are using these tools for their homework, they recommend it [REDACTED:age] their colleagues, and we also know that they prefer using these tools over a human tutor.

10. 1:49

    So this is not new. Uh, we are right now dealing with the first AI generation. These children have been growing up with AI since twenty fifteen. Half of households in US had some sort of voice assistant.

11. 2:04

    And I started researching this in twenty fifteen, um, at MIT, and basically showed that youth perception of voice assistants, chatbots, smart toys really influence how they interact with these devices, how they learn from them, and how they perceive them.

12. 2:22

    So overall, they perceive these AI devices friendly, um, but they also have [REDACTED:age] different perception of the intelligence of the devices, and age plays [REDACTED:age] hu- huge role. So younger kids, we're talking four [REDACTED:age] six and [REDACTED:age], are much more skeptical of how smart Google Home is or Alexa.

13. 2:40

    And older kids, like, the moment they start going [REDACTED:age] school, they all say, "Voice devices are smarter than I am." And we're just talking about voice assistants. We're not talking yet about generative AI tools.

14. 2:51

    So how can we change that and, and why does that matter? It matters because their perception of how this technology works influences why-- what they expect, and their mental models is influencing the type of queries they're gonna ask, how much they trust the answers.

15. 3:07

    So we need [REDACTED:age] cultivate AI literacy and critical understanding of this technology. [REDACTED:age] that end, I built this open source and free platform called Cognimates in twenty sixteen. It expanded Scratch, which is the largest platform for coding for kids, and it ba- basically allowed children [REDACTED:age] do programs like this.

16. 3:26

    Can someone guess what this program does? [muffled voices] Yeah, it's a game. It's a hide and seek game, right? So she's like, she programmed a hide and seek game with the robot, uh, and if she puts a loop, she can run around the room, and the robot is constantly gonna try [REDACTED:age] find her.

17. 3:43

    Now, this is the first step. Um, it's using, like I mentioned, block-based, uh, programming language, expanding Scratch. And at the time, like, it allowed kids [REDACTED:age] not only program their smart lights, their voice assistants, but also train their own custom models.

18. 4:00

    So they can train models with examples of images or examples of text, and then use those custom models in their own games and application. So for example, here, like, this student trained a model [REDACTED:age] distinguish between unicorns and narwhals.

19. 4:16

    And then not only it gets a prediction when it plays with the game, but it also gets the confidence level. How confident is his custom model that the drawing is a unicorn?

20. 4:26

    And we see the confidence is pretty low. So they made all sorts of things, like looking at what's in their food, uh, trying [REDACTED:age], like, uh, program games like rock, paper, scissors, uh, get, like, uh, the robot [REDACTED:age] talk like Shakespeare.

21. 4:43

    And this was used all over the world. It's translated in, uh, more than thirty languages. And, um, the good news is that we evaluated this [REDACTED:age] see how it increases that critical understanding of AI and how it helps with AI literacy.

22. 4:58

    So [REDACTED:age] do that, um, uh, I did a longitudinal study in public and private schools where we asked questions of what kids think about AI before, then we allowed them [REDACTED:age] engage in AI learning activities, and then we asked the same questions at the end.

23. 5:13

    And what we found after they learned how [REDACTED:age] do text training, image training, smart home programming, um, is that they became much more skeptical of the AI smarts. Like, in the beginning, they would say like, "Yes, you know, Google Home is smarter than me," or, "This model is much better than me."

24. 5:31

    And after they learn how it works and how [REDACTED:age] train it, they were not so sure it's smarter than they are. And I'll show you a quick video [REDACTED:age] see how that went.

25. 5:42

    So we were programming robots. You could play rock, paper, scissors.

26. 5:50

    You could rock, paper, scissors into a camera, and it will compute it for the robot's turn, and then the camera will go on the robot's turn. And it's, uh, rock, paper, scissors.

27. 6:03

    Shoot.[gentle music]

28. 6:05

    You do get better at practically anything. It's like us. We might not know everything at first, but as you keep trying, you get better.

29. 6:15

    Everyone has heard about machine-based learning or AI learning. And the sort of question [REDACTED:age] ask for a lot of the more tech-savvy parents was like, "Go for it."

30. 6:23

    Technology is gonna be a huge part of their lives, much more so than my life. If it's scary for some people, this AI technology, I totally get it. But as a parent and as a teacher, I thought it was really important because these are skills that twenty-first century kids need [REDACTED:age] have.

31. 6:39

    When my dad was young, he bought a car and took it apart [REDACTED:age] see how it worked. So you teach people that young how these things that grown-ups mostly program, how it works.

32. 6:56

    So as I was saying, AI engineers in the making. Um, and this, this is the, uh, significance difference, like, [REDACTED:age] their perception of the smarts of AI before and after doing these learning activities.

33. 7:09

    So how did they... Why did that happen, right? Like, why would-- why did they became more skeptical, more critical, and also more literate in how [REDACTED:age] read and write with AI?

34. 7:19

    Is because by providing this platform and allowing them [REDACTED:age] tinker and form hypothesis and test them, we basically allow them [REDACTED:age] engage in the scientific process, just like researchers do, just like we do, right?

35. 7:32

    But we needed [REDACTED:age] have the right sandbox, the right platform for them [REDACTED:age] be able [REDACTED:age] quickly tinker and quickly iterate. So kids are not alone in learning this.

36. 7:41

    Parents need [REDACTED:age] learn too. Teachers need [REDACTED:age] learn too. And we've seen during the pandemic, when kids were stuck at home with, with parents, a huge opportunity for them [REDACTED:age] learn together.

37. 7:52

    So I'll show you, uh, one of the early demos of Cognimates.

38. 7:59

    Oh, the audio is not working on this one. I'm not sure why. Basic-

39. 8:04

    There you go. You did it. No, I need you [REDACTED:age] help me ask a question. For that, we'll need the Ask block. See if you can find it. [popping]

40. 8:20

    Awesome.

41. 8:21

    So the thing that you're programming is kind of collaborating with you [REDACTED:age] teach you how [REDACTED:age] program it, right? Just imagine applying that [REDACTED:age] any of the chatbots we have today, right?

42. 8:31

    Like, when you're not happy with the answer, or maybe the answer is not age appropriate, or you wanna teach... You also wanna teach something [REDACTED:age] the model about your language, your culture, weird facts that you're interested about.

43. 8:44

    How do we do that, right? Um, so I did another study, uh, where I-- this was with kids and parents in ten different states in US over multiple weeks, where we wanted first [REDACTED:age] learn, how do we design a copilot for programming for families?

44. 9:01

    So before we start and build it, like, what do they want? What works and what doesn't? So what we found was that some of the things that kids and parents liked the most was [REDACTED:age] generate coding ideas with an AI friend.

45. 9:15

    Like, if they had a copilot in, in Scratch. Um, and this was very, very helpful. Here are some quotes. Um, because, like here, like, one of the participant says, "Most people would like coding with AI friends because one of the hardest parts of your project is when you start, you run into, into a wall because you're out

46. 9:33

    of ideas." So the AI friend helped with that. It also allowed them [REDACTED:age] express and elaborate their ideas in code. So if they had an idea for a game, like, "I wanna make, uh, the bear kind of jump over the hedgehog," but they didn't know how [REDACTED:age] do it, it would kind of help them, um, find the,

47. 9:52

    the right code constructs [REDACTED:age] do it. And more importantly, it supported their creative coding iden-identity. So it wasn't a bot that was making all the coding. They were doing it.

48. 10:03

    The bot was just helping them when they were stuck. So this was very, very important. It encouraged kids and parents [REDACTED:age] work together, which is not always easy, right?

49. 10:13

    Like, uh, one of the things I discovered, I've been working with kids and families for long time now, since two thousand fifteen. Um, it's not always easy. So actually having like a third moderator [REDACTED:age] be like, "Oh, what does mommy say?

50. 10:25

    What does daddy think? Uh, take turns. Try this," really help with family joint engagement. Also, it doesn't always work, right? Sometimes it's too distracting, and it was very important [REDACTED:age] enable families [REDACTED:age] shut it off.

51. 10:38

    Maybe they wanna do the game alone. They wanna do the coding alone, so they could stop it whenever they wanted. If you have multiple siblings that fight over the laptop, it doesn't really...

52. 10:48

    It cannot help with that. Uh, or if the compli-- if the concepts were too complex, it was not able [REDACTED:age] scaffold it always, like break it down, so parents were very helpful [REDACTED:age], [REDACTED:age] help.

53. 10:59

    So after understanding, like, what are the core things that families want from co-creating and learning how [REDACTED:age] program with an AI friend, I went and basically evaluated all the generative AI models [REDACTED:age] see if they could do that, right?

54. 11:13

    So Scratch, for Scratch, like, top generative AI models are pretty good at generating explanations, giving, like, ideas or questions [REDACTED:age] help, uh, kids and parents, like, explore, like, and test, like, new games.

55. 11:27

    And this was published. We created a benchmark as well for, for measuring this.

56. 11:33

    And this is just an example of what the future of education with multimodal AI could look like. Um, if it's applied [REDACTED:age] Minecraft, [REDACTED:age] games, [REDACTED:age] physics simulations, science simulations, it can become a creative sidekick, right?

57. 11:47

    There are a lot of people who love [REDACTED:age] build things with their hands. What if I could get ideas, like, by taking pictures of flowers I like and colors I like, and it gives me ideas and helps me, like, generate 3D models and that I could afterwards print and paint?

58. 12:03

    Um, or I'm into knitting, and I wanna use a generative AI model [REDACTED:age] inspire my knitting projects. It can also be a learning companion and a coach. It can help with math.

59. 12:13

    So together with Nancy Otero, we created the first benchmark for math misconceptions [REDACTED:age] show what are the most common math problems that kids have in K through 12 and evaluate how good are top of art generative AI models in identifying these misconceptions when kids talk with a chatbot.

60. 12:32

    And I put a link [REDACTED:age] it if you wanna download it. So I am here [REDACTED:age] invite you [REDACTED:age] think about AI engineering and AI tinkering for all ages, and how do we go from my experiments [REDACTED:age] Cognimates [REDACTED:age] things that people are doing and tinkering with Hugging Face and make sure, like, we open up the space

61. 12:50

    so we use AI not just [REDACTED:age] teach, but we actually use AI for people [REDACTED:age] learn how [REDACTED:age] tinker and learn by playing and learn by doing. So I like [REDACTED:age] do what I preach, and I'm gonna show what I tinkered with AI last night.

62. 13:05

    This is-- These are very fresh demos. So this is using the latest, um, Gemini API,

63. 13:13

    and I have three demos. Let's hope they work. Uh, let's start with the science one.

64. 13:27

    And I was hoping [REDACTED:age] draw in real time, but I don't have a table, so luckily I have some, some drawings, and we'll see, we'll see how well this works.

65. 13:38

    So I have a drawing. Oops. A scale with a weight on each side. What would happen if you add another five kilograms?

66. 13:55

    So it's asking me questions based off on my drawings, and then I can make a new drawing that has, like, ten kilograms and ten kilograms and see if that gets better.

67. 14:03

    Um, let's try another one. Water and CO2, what happens if it gets mixed? Oh, I need [REDACTED:age]... So imagine I have a webcam, and I'm like a table, and I'm drawing in real time, and we could play with it, but it's very interactive.

68. 14:24

    Let's try this one. The Earth is being hit by something. [laughs] Hopefully not.

69. 14:34

    Let's, let's add one more arrow and see what happens if we do that.

70. 14:47

    Oh. [laughs] Ah. [laughs] That was fun. So it finally understood it was the moon. Let's play with the math one.

71. 14:56

    And... Solve the

72. 15:20

    expression inside the parenthesis. Okay, so I have one where I did.

73. 15:34

    Okay. Solve the multiplication with the parenthesis. And let's assume I've done that too, and I have the next question.

74. 15:45

    I need a better background for this demo, that's for sure. [laughs] Uh, the first step is [REDACTED:age] simplify. No, no, no, go back. Okay, so any number divided by itself equals one.

75. 15:54

    But you see, it doesn't give me the answer. It just give me a question so I can keep trying, right? And learning. Um, let's try one more, more complicated.

76. 16:06

    So okay, four on the axis. Oh, I was hoping it would give me a better question. Yeah, so the last one is the one that is encouraging curiosity. So this one...

77. 16:35

    What is the lady doing? Okay. Uh, what are the colors on the flag? What shape is the star?

78. 16:46

    Oh, it asked me things about Jordan. Um, let's see what it does with apple.

79. 16:55

    It has... Can you-- Do you know what apple this is?

80. 17:04

    Uh, let's see. What does the apple smell like?

81. 17:10

    Uh, had a nice origami thing as well.

82. 17:21

    Okay. I don't know where the origami went, but...

83. 17:27

    So you sort of get the gist. Um, these, these are like... I don't know if... Do you want me [REDACTED:age] draw something, or do you want [REDACTED:age] ask one of a question of the science or math or objects?

84. 17:37

    Any, any requests from the audience? Don't be shy.

85. 17:50

    Yeah? What, what should we ask? No? Okay. Well, uh- You

86. 17:59

    gotta ask. [laughs] Can you ask it [REDACTED:age] solve a system of equations? Excuse me? A system of equations. A system of equation. Yeah. Can you tell me what [REDACTED:age] write?

87. 18:09

    Sure. Uh, two X plus seven- Two X plus seven-

88. 18:15

    Equals two ... equals two. Thank you. Let's try it.

89. 18:32

    Let's see if it does well with my, uh...

90. 18:36

    Subtract seven from both sides of the equation. Mm, not bad. And now if I do that... [laughs] [laughs]

91. 18:57

    Divide both sides by two and so on and so forth. Um, now, the cool thing about this, like, I made the code open source and template so you can play with it too.

92. 19:06

    It's less than a hundred lines. You just need [REDACTED:age] create an API key, which is free, and you can create your own instructions. And hopefully, I inspired you [REDACTED:age] think, like, beyond of chatbot interfaces and delegating instructions and delegating, like, questions and think more in, like, a tinkerer and think about how we could put these tools in

93. 19:30

    the hands of young people because they are the future, and they need [REDACTED:age] learn about this technology as well and how it works. Um, I think that's my time.

94. 19:38

    Uh, all my research is on my website, and I put a QR link for that as well, and I look forward [REDACTED:age] your questions afterwards. Thank you so much. [outro music]