Tools for the Next Generation of AI Engineers

[on hold music] Thank you so much.

Hello, New York. Uh, how are you doing?

Good. [laughs] Okay, so I know you've been hearing a lot of amazing talks, and before I get started, I wanted to see a quick show of hands. How many of you are here for the first time at AI Engineering Conference?

Wow, that's amazing. How many of you are international, outside of US?

Incredible. Thank you for coming. Um, so today I'm gonna talk about how do we open up this conference and how do we open up this knowledge to the next generation of AI engineers, which are young and, uh, well, much, uh...

start much earlier. And why does that matter? It's because 70% of AI users are actually from Gen Z. Um, and we've seen the potential of multi- multimodal AI to transform education.

We know that students are using generative AI tools for their homework, and I think they can use it in a much more interesting way. We also know that in many cases, they prefer it, um, to, uh, human tutors.

And I want kids to actually be part of the engineers and the designers that create the tools for them to use. Um, and I turned to Scratch. I don't know, how many of you know Scratch?

Wow. Uh, there are over 100 million children using Scratch worldwide. Scratch is a pr- a platform, it's free, it's open source, for coding for kids, it's visual programming. It was d- developed at MIT, and I was part of that lab and, um, during my master's, and that's in 2015 when I start working on Cognimates.

Cognimates expands Scratch to actually allow children to learn about AI by building games, training their own, uh, AI models, and programming hardware as well. So it has these kind of blocks, they look like Lego blocks, and they can put them together in order to create their programs.

Does anyone want to guess what this program do?

Any guesses? So Tiana here, who was [REDACTED:age] at the time, she, she wrote this program for the first time and then played with the robot, um, using the program for half an hour, which was super fun.

So it's a hide-and-seek game, right? So she would, like, run around the room, and the robot would turn around and say-- Every time it would detect a person, the number of people was, like, higher than zero, it would say, "I see you."

There is a pro- a problem with the program, though. Can you spot it?

How many times are we gonna play if we have it like this? Yeah. One. Yeah, so what do we need, Nick? Uh, we need a loop. We need a loop.

Exactly. Awesome. Um, so the time to fun and the time to play, um, is very short in Scratch, which is why I used it when I built Cognimates. It has this library of blocks, a coding area, and a stage.

And I'm gonna show you a quick video of how kids use it in order to learn more about AI. [upbeat music]

So we were programming robots. You could play rock, paper, scissors.

You did rock, paper, scissors into the camera, and on cue, you did one of the motions, and the camera did one of the motions, and it's like rock, paper, scissors, shoot.

The computer gets, like, better as you play the game. 'Cause like us, we might not know everything at first, but if we keep trying, we get better.

Everyone has heard about, like, machine-based learning or artificial intelligence, and there was a sort of no questions asked for a lot of the more tech-savvy parents. It was like, "Go for it."

Technology is gonna be a huge part of their lives, much more so than my life. If it's scary for some people, this AI technology, I totally get it. But as a parent and as a teacher, I thought it was really important because these are skills that twenty-first century kids need to have.

When my dad was young, he bought a car and took it apart to see how it worked. So you teach people that young how these things that grown-ups mostly program, how it works.

So we did this in 2015, 2016. Uh, these AI engineers started very early. And at the time, they were training these custom models with basic classification models for images and text.

Um, but they also had access to the entire library of extensions that we built for them, so they could use off-the-shelf sentiment analysis, image classification. They could program their voice assistants because they realized voice assistants were really limited.

So they could program the voice assistants to remember things about them and their preferences, um, program micro bits, robots. And this is what the training page looks like. Um, they can drag and drop examples of images.

So for example, one kid wanted to, um, make a game about unicorns and narwhals. So this was, like, what his training data looked like. And then this is what the program looked like.

So he can choose his custom model, unic- unicorns versus narwhals, and then show different drawings into the camera and see what the model predicts. And moreover, it als- he also or she can see-- they can see the level of confidence of the prediction, which really helps them understand, "Oh, yes, this-- in this case, it guessed that my

drawing was a narwhal, but the confidence level is very, very low. What do I need to do? I'm gonna go back to the-- my training and add more examples of images that are hand-drawn because most of my images are cartoons."

So it really creates this AI li- literacy, data literacy, and demystifies everything that kids, uh, learn about AI or hear about AI. That it's evil, Terminator, or all sorts of things, right?

Um, and- They built projects that went across different domains. So it was a project where they could look at things in their food by transforming a webcam into a microscope.

They could build games like the rock, paper, scissors that you've seen in the video, or a literature program, uh, where you speak and it analyzes what you say and s-- uh, it's seeing if it's in the style of a s- famous writer or, uh, different other, uh, types of styles.

And what's important, like, I tested this with kids in public, private, uh, s- uh, schools, and community centers, and what I found is that kids are actually like little scientists.

So if we give them the right tools, they engage in the scientific process. They formulate hypothesis about how the model works or how the robot works, and then they test those hypothesis, and they refine their understanding in the process.

And it's the same with the model training. So we need to create tools that enable them to engage in the scientific process as fast as possible, and create tools that are fun and sticky that they want to use.

And why does this matter? Um, before, uh, allowing them to program with Cognimates and train their own models, I actually asked them questions about voice assistants and other, uh, smart toys and smart, uh, robots at the time.

And I asked them, "Do you think it's smart? Do you trust it? Um, do you like it? Is it friendly?" And I asked the same questions about those technologies, we didn't have ChatGPT at the time or Gemini, um, at the end of the study.

And what I found was that there was a significant difference in the intelligence attribution after they engage in this process of learning how to train a model, learning how to program it, understanding why the data matters.

Um, so it does make a difference in demystifying the intelligence that we talk about. And this platform is used around the world. It's actually translated in fifty languages. And I realized after I worked on this, that it's not just the kids.

Like, the pandemic came, and a lot of young people were stuck at home, um, so I had to, like, really think about how do I create and how do I work with families.

So started to do, uh, a lot of other experiments. And I figured, like, what are the type of tutors or, uh, games or, um, platforms that families could use when they're at home and maybe they wanna learn how to code with their kids?

And I'll show you an early prototype.

Hi there. I would like to know your name, so let's do a program that allow me to learn it. Let's start with the green flag block.

There you go. You did it. Now I need you to help me ask a question. For that, we'll need the ask block. See if you can find it. [pop]

Awesome.

So in this case, they're learning how to program a robot. It was a Jibo robot, and the robot itself, it's participating in the process. So you're having, like, this reflective, uh, conversation with the thing that you're programming, which is pretty cool.

Uh, and we could do that much more than that now. Um, but because not everyone can afford to buy a robot or, like, it's not, you know, a required thing, I wanted to build something similar, like a pair companion for co- programming that works in the browser.

So during the pandemic, I start doing these design studies with families in ten different states, uh, in US. Very different backgrounds, very different ethnicities. And this is very important, and I wanted to highlight that here.

Before building the system, I mocked the system. So we didn't have a functional copilot or a functional assistant. I was the AI, um, and it was a Wizard of Oz study.

Like, I, we were mo-- The kids did not know initially, we told them afterwards, that they're interacting with a person that... But they would interact via chat, and I really wanted to understand, what is it that they want?

What kind of supports do they need from a pair programmer when they code on Scratch with their parents? And what I found was that they really wanna generate coding ideas.

They don't want the copilot, uh, in Scratch or in Cognimates to do everything for them. They wanna kind of brainstorm, like, "Oh, uh, what if I wanna do a game about bears?"

Or, "What if I-- I'm into soccer, like, give me some ideas." Um, that was a really big one. And here are some quotes. Like, uh, one of our participants who's twelve said, "Most people would like coding with AI friends because one of the hardest parts of our project is when you start and you run into a wall.

You are out of ideas." So it really helped with the ideation process. It also helped them kind of express and elaborate their ideas. Like, if they would play a, a Pong game, it would ask like, "Okay, so how do we make the ball move?"

Or like, "How do you make it move faster?" Um, so it was very helpful in that regard as well. And it supported their creative coding identity, right? So another quote, "I like it because sometimes when you code, it gets frustrating.

When you finally get it to work, it's so good that it lets you feel good. It's good when you have someone that says to you, 'Good job.'" And what's interesting is that the kids participating in these studies spent double the amount of time programming that they would spend normally, and I, I got that from parents.

And didn't always work. Sometimes, like, uh, uh, the parents were needed in the loop, like if it was too distracting or if it wasn't able to moderate turn-taking between siblings.

Uh, or if the [laughs] yeah, you know, uh, agents have limitations, AI has limitations. Or it wasn't always able to explain the most complex concepts, like the, uh, in Scratch, there is this thing called clone when you create multiple instances of the same object or broadcast.

That was harder to explain. So after doing this first design study and identifying, like, what are the core features that kids and parents want, um, I created a, a evaluation benchmark.

So I actually had over a hundred cases of Scratch programs And I would run this against, like, different state-of-the-art models and see how good these models are at explaining the Scratch code, at explaining it with learning exercises, at debugging it, at c- generating ideas.

And the results were very promising. So, um, the next thing was to build it. And this is, uh, the first time I'm showing these results. I just finished running the study, so it's very o- off the press, uh, very fresh, uh, from the, from the oven.

And, um, uh, I, I just tested this Cognimates Copilot with young people, um, 18 young people from 11 different countries in different languages. And it was, it was kind of cool.

Like, it's very simple. It's the code editor, and it has this AI chat. Um, it's sending a message to a web server and then gets a response. I'm, I'm using and evaluating different models, including a fine-tuned model on Scratch projects, and it can also generate assets, art, like images for their games.

And here's an example from a session with a kid from Mexico. So all the session, like, was done in Spanish. I speak many languages, so I could do the session in different languages, and so does the copilot.

Um, and what I found in analyzing these sessions, uh, from il- 18 kids is that the copilot provided all sorts of supports. And it was conceptual support, design support, positive encouragement, platform nav- navigation.

There were instances where it failed. Um, and I also saw lots of instances where the kids would refuse the suggested help or the suggested ideas.

And here's an example of code support. This was a student from Jamaica, and he had never programmed in Scratch before. So he was able to go from zero to actually having a fully functional program with the support of the copilot.

And the co- the copilot was actually very helpful, not only in giving him ideas, but also helping him understand how to navigate the platform for a very, for the very first time.

Like, "This is where you find the loop, uh, block, and this is where... how you create variables," or... Um, so it was very helpful for people who are new.

For people who are advanced, it was also very helpful because it would generate assets that they really liked. Um, it would give them ideas, like, for how to refactor.

It didn't call it refactor, but, like, improve the code, um, or how to add new features or new levels. And it was interesting because kids would use it in a lot of ways I didn't predict.

Like, besides, like, generating the, the background in this game, also, like, for they wanted to get names, ideas for names for characters and ideas for plots. Um, and this is a great example, like, where a student is actually pasting, uh, an image of the code, and he's asking for an idea, and he actually doesn't like the answer.

So he said, like, "I don't wanna do that type of movement." And then the copilot says, "No worries. It's your game. Have fun." [laughs] Right? So we wanna have tools that are actually prioritizing and encouraging young people's agency in this process.

Some lessons learned. Uh, uh, to prioritize user's agency, to balance the support and the challenge that we give them. By default, the copilot does not give the answer. Initially, it ask, asks questions, and if the student is really stuck after asking the same question three times, it would give a hint.

Um, to see these agents as motivators and starting point, right? Like, the effect of the blank page when you start to write. The effect i- in, in Scratch is called cold start.

We see a lot of students that go to the platform, and they really don't know where to start. So it's very helpful for that too. And we learned that it was important to allow for flexibility and customization.

I had kids that really wanted to use voice and other kids that did not wanna use voice. They just wanted to type, right? I had kids that said, "I always want it to give me three ideas," and other kids that was like, "No, just one idea is good."

Um, so everyone wanted different things. And design agents that support, um, creativity is key too. So a lot of the participants told me they want the agent to be able to go and move the blocks with them, and they wanna have, like, see simulations of agents programming.

Like, "How would an agent build a Pac-Man game? I wanna see, like, how five agents would collaborate to build this Asteroid game." Um, or, "I want when it generates the assets, instead of giving it to me in the chat, like, to have it directly on the stage."

So the next, the next phase in prota- of the, of the prototype is to actually hack the OS entirely and integrate the, the agent at all different stages, uh, of the UI.

And then support multimodal AI capabilities, like, including, like, sound generation and maybe reaction to camera stream. Um, and then the, the part that I... was very important is when it doesn't work or it cannot do what they ask of it to do, um, it should tell them why, right?

Like, it should be like, "Okay, I cannot generate this type of images because my training set does not include it," or, "I can only give you answers about Scratch because that's what my prompt is," or, um...

So it should be really transparent, uh, and explain its limitation in order to set the right expectations. So I'm gonna show you a quick demo. Let's see if it works.

And this is just a prototype. And I learned that I'm giving this talk two hours ago, so please be kind. Um, so-

So-

Oh, oops. How do I get it there? Thank you.

Yeah.

Okay. So when... Let's see. I'm gonna reload it. Looks like this. So if you come here and you don't know what it can do or not, we could just say hi.

Uh, it's gonna be like, "What do you wanna work on Scratch today?" What should I say?

A racing game. I love that. Racing game. Give me

some ideas. Turbo boost button, extra speed. That sounds great. Um, but the cool thing is that it actually integrates with Scratch, so I can go and get, like, any, uh, project from Scratch.

Like, people build really crazy stuff on Scratch. So let's say, like, they're, they build some OCR programs. Uh, this is a bit slow, but people are actually building OCR implementation in Scratch.

Um, and if I find a project that I like, uh, if this works, then I could actually load it in, in my Scratch. I have one that is downloaded, so I'm just gonna load that.

And then let's see. [laughs] So first let's see how it works. Like, I can draw any number here

and then ask it to recognize it. Okay, thinks it's two and eight. Um, but this is a pretty complex program, right? So if I don't know how something works, like, I can actually do a screenshot.

Oh, I did not expect to have two screens. Um, I, I can do a screenshot and then, like, attach it to the chat and it would explain to me what the, what the code in the...

See if I can get it fast enough.

Yeah, I can't get the screenshot from the other tab, but you got the idea. Um, so that's kind of what the program is, and let me go back to this.

And, um, it's free, it's open source. Like I mentioned, there are a lot of new features coming. I hope you can contribute and give us feedback or share it with your y- young friends.

Um, and the reason why this matters is because AI literacy is now actually part of the law. I don't know if you know, but this was passed earlier this month.

It's part of the EU AI Act, and it basically says that all providers and deployers of AI systems should take measures to ensure, to the best of their extent, sufficient level of AI literacy, uh, of their staff, but also of the users of their products, right?

So we need to, in order to ensure this AI literacy, we need to start early, and, um, that's, that's what I'm hoping and trying to do with my work.

Um, if you wanna learn more, uh, there are lots of papers and studies, uh, in, about AI literacy, about AI education, uh, work done on other domains, math misconceptions and science.

It's all on my website. Um, thank you so much. [clapping] [outro music]