TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM

Talk transcript

1. 0:00

   [upbeat music] Hey, my name is, uh, Cormac Brick.

2. 0:16

   I work, um, on Google AI Edge, which is a way of kind of bringing models to the edge. This is something we use internally for our own products. It's also something we make available as open source products.

3. 0:28

   As part of this, we also work really, really close with the Gemma team because they publish a lot of models that are targeted to edge devices as well. Um, yeah, quick bit of background on me.

4. 0:38

   Um, yeah, I've been focusing on edge AI for the last, like, wow, like, maybe 10 years at, at this point. Um, so started off in 2016 with, like, running a, uh, Google Net on a USB key hardware accelerator plugged into a Raspberry Pi at, like, NeurIPS 2016.

5. 0:56

   So yeah, it's been a lot of fun over the years. Uh, then joined Intel, w- worked on... Led the architecture for the NPU that goes into all of their laptops these days.

6. 1:04

   Three years ago, I moved to Google to work as tech lead on edge AI here. Uh, yeah, and it's been great fun. Like, been a crazy few years, as you can imagine.

7. 1:13

   Um, but yeah, happy to share, like, both where we're up to and also give a kind of overview of, like, as part of that, we can also k- kind of cover the story of, like, where is, um, where is mobile, mobile AI up to today, right?

8. 1:26

   Like, what's the state of the art? What are the different patterns we see for model deployment? So the two things that I want to focus on in this talk are, uh, tiny LLMs and agent skills.

9. 1:37

   Um, so tiny LLMs are very, very small models. Um, agent skills are now possible on device with larger models, uh, are needed to make those kind of work fluently on device.

10. 1:47

    But both are kind of exciting new directions, um, that have only recently been made possible. So as recently as last week when we launched, um, uh, Gemma 4, um, with...

11. 1:58

    Uh, the DeepMind team launched that. We supported them with a kind of Android and iOS app. That's kind of opened, uh, new possibilities for things we can do on mobile.

12. 2:06

    So we'll also kind of deep dive into that, um, as well as then looking at what are the types of things we can do with tiny models. So kind of two halves of the presentation.

13. 2:14

    Okay, so that's the intro. So yeah, this is kind of what I was saying. [laughs] Uh, so first give an overview of, like, hey, what's, like, AI on the edge, like, looking at what we look at as small language models and tiny language models.

14. 2:29

    Uh, then taking a look at Gemma models, just because that's, um, uh, pretty topical, uh, and the types of performance we see for Gemma models on various types of edge devices, because that's kind of what our team does.

15. 2:41

    Um, then looking at agent skills, which we built on top of the latest generation of, uh, Gemma models, um, uh, that can run on both Android and iOS, as well as many other platforms.

16. 2:52

    And then, um, second half, we're going to look at tiny models and how, um, like, how you can fine-tune and deploy a tiny model to edge devices. Um, and then finally, I've an example of an app that our team built, um, that...

17. 3:09

    Like, a real app, [laughs] uh, using... Not just an example app, using, uh... That's built using tiny LLMs based on Gemma technology as well.

18. 3:18

    Okay. Okay, so firstly, yeah, I guess folks who are... You guys are already in this room, so you're probably interested in edge AI. This probably goes without saying, but, um, yeah, like, there's a lot of benefits to running on the edge.

19. 3:31

    There's, um, like, latency or UX improvements for some really sensitive in the loop, um, things like live voice translation, for example. Uh, that's something our team shipped on Pixel last year, where you can do a live voice translation.

20. 3:46

    Very challenging to do that with the required latency using a cloud service. So being able to do that offline and on the edge is key to be able to meet latency or user experience requirements.

21. 3:55

    Um, privacy, that's, um, uh, that's definitely a thing as well. Like, we, um, do some, uh... Yeah, uh, there's, uh... People use AI within, like, messaging apps, for example, and people like their messages just to stay on their phone encrypted, fully private.

22. 4:12

    So that's a good kind of example of where we're seeing LLMs getting deployed, is assisting in some of those types of use cases. Ooh. Okay. Where privacy is, uh, really important.

23. 4:23

    Um, offline use. Yeah, that's kind of obvious. And then savings. Um, yeah, like we see this being increasingly relevant for laptop users, where you see a trend towards folks at, at least experimenting with, um, uh, experimenting with, um, small language models to do some types of tokens they may do with a desktop agentic workflow they have going

24. 4:44

    on. Uh, so some folks are interested in exploring savings as well. Uh, yeah.

25. 4:51

    Okay, so this is, like, what we do as, like, our team. We do enable people to build apps. Uh, we do, uh, MediaPipe, uh, LiteRT-LM, which is a LLM runtime that works on mobile and edge, and then LiteRT, which is just a standard inference framework.

26. 5:07

    This was a long time ago known as TensorFlow Lite.

27. 5:11

    That is the types of things we've used that over the years in, is, like, it ships in photos in... Unblur uses MediaPipe in photos. YouTube Shorts, uh, if you've used any of the funny effects in YouTube Shorts, they're also things that our team have built, uh, for these, uh, along with these, with the YouTube team,

28. 5:29

    um, using MediaPipe and various deep learning tracking models under the hood. Um, and that's also built on the same, uh, MediaPipe and LiteRT technology. And then, yeah, we run this on, like, all Android phones today will use this as part of system services, um, or, you know, third-party apps building on a version of the stack that ships

29. 5:50

    with Android. Um, but our stack also, like, runs far beyond Android. Um, so we also run... Even it being available, um, as a system service in Android, it's also you can take the same TF Lite file and ship that to iOS, macOS, Linux, Windows, web, or IoT devices just from that same file.

30. 6:13

    Um, so yeah, that's kind of why deployment is helpful. Uh, one caveat there. That same file deploys on CPU and GPU. For NPU, we need to do some special compilation, and you end up with a special NPU file.

31. 6:25

    Um, but certainly for, even for, like, the Gemma models we published last week, this is, uh, true that one file can work in CPU and GPU across a wide set of devices.

32. 6:37

    So then, yeah, some, some of the things we're seeing for LLMs on device is kind of privacy centric stuff, uh, voice agents and local agents, uh, with tool calling is a very popular workflow.

33. 6:50

    And then, yeah, within our stack we... LiteRT-LM is the, uh, thing that we'll look at a bit later that helps us run, um, tiny LLMs on device cross-platform, and it's fast because we support all of the different hardware accelerators.

34. 7:05

    Oh, yeah, so this is kind of important, uh, concept is we see two trends happening today. Uh, one is we see kind of system level gen AI. So for very, very large models, the way these tend to turn up on mobile phones isn't that, you know, when you launch a single app, it will download a 4 billion

35. 7:26

    parameter model, uh, just to, uh, you know, just to help you find a good restaurant in, you know, uh, in whatever app you're looking up, looking in, right? Uh, instead, the trend is to build larger models into the OS.

36. 7:40

    So we call this, like, system level gen AI. These models tend to be

37. 7:45

    in the kind of 2 to 5 billion parameter range, right? Like, it depends on the OS and, um, whatever. But, like, this is one choice that we see both the Android team, who we work closely with, and the, um, Apple Intelligence team.

38. 7:58

    We see kind of similar choices being made by both mobile OS vendors where there's a central model built in, and that is, um, like on Android that's called AI Core, and there's things like summarization APIs and, uh, an increasing set of APIs available for developers, including a prompt API where you can use that API.

39. 8:16

    Um, and that's available on kind of premium Android devices and, uh, premium Apple devices as well. Or, or the... Like, obviously Apple has their own Apple Intelligence thing. But as a trend, this is kind of worth, um, uh...

40. 8:29

    This is really relevant, right? So if you want a, um, if you want to leverage a, a built-in model, this is a great way to go. Um,

41. 8:39

    so then the other trend we see is, like, in-app gen AI, which is where the kind of tiny LLMs or, um, TLMs as we're calling them in this presentation are more, uh, are more relevant, right?

42. 8:51

    Uh, so system gen AI, generally you customize that via prompting, um, or via skills as we're going to see later. Um, um, and it's a foundation model preloaded on device.

43. 9:03

    In-app gen AI, generally these are custom to tasks. They're loaded with the app or with the webpage. We also deploy some of these, uh, on the web. And generally they're targeted at kind of like wider reach.

44. 9:13

    So this may work not just on premium devices but all devices because apps, uh, reach is really important for a lot of the app, application developer teams that we work with.

45. 9:23

    Um, and these, um, yeah, surprisingly you can get LLMs to, uh, if you fine tune for a single task, we've seen really, really strong performance on, you know, simple tasks like, you know, summarization, transcription, or kind of voice to action type things.

46. 9:39

    We can get really reliable performance from models in the, um, uh, really reliable performance for models in the, like, 100 to 500 million parameter range depending on the complexity of the task, right?

47. 9:53

    Um, so a good example here is we launched Function Gemma, um, in December, and that's a, was a 270 million parameter model that was dedicated for function calling. And we're able to show that, uh, doing, uh, like 10 dif- doing voice to function calling for 10 different, uh, functions that were relevant to kind of Android developers.

48. 10:17

    And our internal evals, we did an internal eval set that kind of reached over, um, it's like 85 to 90%, uh, kind of reliability just using that very small model which was widely deployable to iOS and Android devices.

49. 10:32

    And that's actually something you can play with in a, a sample app that, that we have that we'll look more at a bit later. So that's an example of, like, a tiny, um, of one use of a kind of a tiny LLM model.

50. 10:44

    Um, but we're seeing more, um, interest from application developer teams now into kind of fine-tuning models to deploy as in-app gen AI. So the difference here is on the left, yeah, you kind of, um...

51. 10:55

    The customization on the left is via prompting or skills. Customization on the right.

52. 11:01

    Like, we would encourage people to do some degree of kind of fine-tuning to make a tiny LLM work. Like, in practice, like, um, yeah, certainly below 500 million parameters that's true.

53. 11:12

    Maybe for 500 million parameters and above, um, we can do more general purpose tasks with, with a model. But for the really, really tiny models at certainly less than 500, in our experience you need to fine tune to get kind of production level reliability.

54. 11:29

    Um, okay. So now I'm going to talk about Gemma 4. Um, uh, and Gemma 4, the, the models that were launched last week kind of fall into that, um, kind of system gen AI kind of category, right?

55. 11:43

    So we can do lots of powerful things with Gemma 4, um, as you'll see. Uh, okay, first we're going to talk about sizes. [laughs] So there's two small sizes which are the first two, uh, which are E2B and E4B.

56. 11:58

    Um, so these have... Uh, E2B is called that because it has It, um, only needs about 2 billion parameters to be present in RAM to l- to run the model 'cause one of the limiting factors we see with these models is how much RAM you need to have in a device in order to run it reliably.

57. 12:16

    Uh, E4B has been optimized to run it with 4 billion parameters on device. Um, there are more parameters used by the model, but the other parameters that the model uses are, um, you could...

58. 12:27

    Like, DeepMind have talks on this later in the week, so you can go there for more detail. But the, the TLDR is the other parameters in the model are used, uh, for kind of per-layer embeddings.

59. 12:37

    So in our runtime, we don't actually need to load all of those parameters. So we need to maintain them, like the, the 2B and 4B, they need to stay resident in RAM.

60. 12:46

    The other ones we typically, like, m- memory map them, and we only need to load, like, one line of the embedding table, of the per-layer embedding table and the embedding table, uh, once in the autoregressive loop.

61. 12:57

    And we actually only need to load a few,

62. 13:02

    uh, a few hundred bytes or thousands of bytes of that, uh, in order to do the next token inference. So as you go through inference, you don't ever end up requiring to load the whole, uh, PLE table into memory.

63. 13:14

    And then, you know, depending on the, uh, OS, it'll do a reasonably good job of kind of, um, uh, you know, evicting memory used by kind of older PLE tokens.

64. 13:25

    Um, so that's why we have this idea of effective. Uh, so the, the smaller models run on... Yeah, you can see on the right, it runs on a variety of platforms.

65. 13:33

    Um, the E2B and E4B models, these models are on the AI Core roadmap, so these models that are available now for experimentation. Um, uh, at the appropriate point in time in the future, the Android team will integrate this into AI Core, and they'll be available, uh, more broadly on a wider set of devices.

66. 13:51

    And there's another, uh, talk next week, or there's a talk here this week or, um, from, uh, Ollie from the AI Core team, who'll probably share more details about exactly what that roadmap looks like.

67. 14:02

    And also there's, uh... Yeah, Omar is gonna do it from the GDM team, is gonna do a deeper dive on, on all of the, the Gemma world. The, the bottom two models, just for reference, these are also relevant for the edge.

68. 14:14

    They're not really the focus of my talk today, uh, 'cause these are relevant for folks to run on laptops. Uh, the sizes have been optimized to run really well on consumer-grade laptops.

69. 14:24

    Um, um, albeit ones that have maybe, you know, 32 gigs of RAM. But, um, yeah, that's what these do. So I'm gonna focus less on these, uh, other two models, even though they're very powerful and useful.

70. 14:39

    Um, yeah. So we can see, like, certainly the E4B model, uh, like both the E2B and the E4B model have excellent, uh, performance on a wide range of things.

71. 14:49

    Um, um, pr- yeah, on knowledge and reasoning. Um, but one of the big step-ups relative to the... from my perspective as a, like, user of the models relative to the last generation is, uh, they've kind of built-in function calling, which is excellent, and they also have built-in thinking.

72. 15:06

    So that combination of kind of thinking plus function calling is what unlocks our ability to now do skills on device. Um, as to... So you can just... A- as you'll see in a second, we can, uh, describe a skill, give it to the model, and the model can just pick it up and, and use it.

73. 15:21

    So that allows us to use that kind of pattern that's proven very popular in the last few months to bring that pattern to mobile, which allows for new types of kind of mobile experiences.

74. 15:34

    Uh, also for the E2B and E4B, these are multimodal, so they support audio, um, image, and, uh, text. Um, the larger models support just image and text.

75. 15:50

    Oh, uh, also the other change from a deployment point of view is this is the first time the Gemma models are released as in just a stock standard Apache 2.0 license, which means they're more usable by more people.

76. 16:03

    Um, again, you'll hear more about this in Omar's talk. Just thought I'd mention it here. Okay. So th- that's Gemma in general. Now just to deep dive Gemma on our runtime and our platform.

77. 16:14

    Um, so yeah, that same kind of picture earlier, [laughs] right? We have a single LiteRT-LM file, which is a LiteRT file with the things like the tokenizer and the other things we need in one package to run a model.

78. 16:27

    Uh, that single model runs, um, uh, across, uh, all of these classes of devices, um, uh, across mobile, desktop, and embedded. And we're, yeah, we're pretty excited to, to do more in the embedded space.

79. 16:42

    There's a lot, particularly with image input, there's a lot of, um, scope for new, um, uh, for, for new, um, IoT use cases. Okay. Bit of an eye chart now, but just [laughs] dig into performance.

80. 16:55

    So, um, I would say this is a snapshot as of today. Uh, this is something we are continuing to work on, uh, both in cells-- both ourselves and our team, and also with various partners that we have, right, across, like, Intel, um, uh, with the Raspberry Pi team and the Qualcomm team.

81. 17:12

    We're continuing to optimize, uh, all of these numbers. But we can see we can do, um,

82. 17:19

    certainly on, you know, for the 2 billion parameter model, we can get really compelling performance on a, like a Pi and Android phone can do thousands of tokens per second on a GPU.

83. 17:30

    Um, also thousands of tokens per second on MacBook or, um... But the, the bottom two rows then are, like, on a Raspberry Pi, we can get, um, about 133 tokens per second, which is sufficient to do, um, kind of simple image analysis use cases with reasonable latency.

84. 17:48

    Um, and also the bottom one is us running on a hardware accelerator. This is a Qualcomm kind of IoT/robotics development, uh, like development platform, uh, that they have available.

85. 18:01

    And there we can see pretty compelling performance as well because we've gone and used the, uh, NPU, which, uh, which gives, uh, a lot better performance. And then, yeah, you can imagine, uh, then we have corresponding performance on the E4b model, um, which works, uh, on, yeah, a wide set of devices as well with, um,

86. 18:21

    yeah, kind of proportionally less prefill and decode performance, right? Given the size and the numbers of parameters we need to fetch. But yeah, broadly they're available on lots of platforms.

87. 18:32

    Okay. So now what can we do with those models? Uh, lots of things, right? Um, so one of the things I wanted to talk more about because it's like net new rather than just kind of show you like our image analysis or audio transcription or audio translation, right?

88. 18:46

    There's a lot of things we've been able to do with models from my perspective for a while that these... that, uh, the Gemma 4 models are, you know, much better than their predecessors at.

89. 18:56

    Uh, but we, um... The thing we're-- The thing that's newest, uh, from my perspective is kind of agent skills on device. So this is, uh... So we have an, an app.

90. 19:07

    Um, I don't know if you guys have seen it. It's available on both iOS and Android, uh, called Google AI Edge Gallery. And, um, this allows you to do lots of different things, like to just do basic AI chat stuff or ask questions of an image or, um, uh, do kind of transcription or translation use cases starting

91. 19:26

    from audio or like audio to function calling type use cases. But the one in, um... Yeah, the second from the left is agent skills that I'm going to do a deeper dive on today.

92. 19:40

    Okay, hopefully this will play. Ugh. Okay. That is very annoying. Sorry. I, I will fix this before we post slides. Um, oh wait. Let me see if this will...

93. 19:59

    Ah, okay. And is the sound going to work?

94. 20:05

    Morning Gemma. Let's log a new mood journal entry with school night plans with-

95. 20:08

    Hmm. Okay. So this is playing but sound is not working

96. 20:11

    I got eight hours of sleep-

97. 20:12

    Oh.

98. 20:13

    And I'm looking forward to hanging out with Amy today.

99. 20:17

    So this is a journal skill app we're looking at here

100. 20:20

     or a mood tracker where you... where it logs mood and sleep and then-

101. 20:27

     Perfect. Analyze the trend in my mood over the last seven days.

102. 20:41

     Okay. So what's happening here is, um, with the mood tracker app, it kind of logs, uh, it logs your moods or observations to a diary-

103. 20:51

     Awesome.

104. 20:51

     And then the LLM is able to go back in and summarize the content of that diary.

105. 20:53

     I'm busy making breakfast. Can you check my calendar for the day and show me in bullet list?

106. 20:55

     What's interesting here is this is just... Okay. This is just... I, I won't kind of repeat this more. Okay. So what's interesting here though is the way this is done, which we're going to look in more detail.

107. 21:05

     This is, isn't a kind of custom fine-tune. It is just us giving a particular skill-

108. 21:12

     Amy wants to meet and must be available

109. 21:12

     ... with a little bit of JavaScript that the, uh, that the model can call, um, to the model. And then through just a free text interface, you're able to...

110. 21:21

     like the model is able to use and pick up that skill. So it's also able to call a... There's two skills act- actually happening here.

111. 21:27

     I believe Amy will pick up her Oscars this Oscar.

112. 21:27

     One is the mood tracker skill, and the other one is the, um-

113. 21:30

     Can you check Wikipedia for the latest information?

114. 21:33

     The other one is the, uh, was the, uh, map skill, and now it's using a query Wikipedia skill here to look for latest information from the Oscars. So versus having a model that's just like pinned in time, it now becomes really easy-

115. 21:46

     Which film won best picture?

116. 21:46

     ... to extend the model with like, uh, with skills such as kind of map lookup, something like, uh, interactive journal-

117. 21:53

     Which film won the most nominations?

118. 21:53

     ... that you can both add, subtract entries to from voice as well as query. Um, uh, and also kind of adding more modern knowledge or relevant knowledge.

119. 22:03

     For breakfast, I'm feeling a little...

120. 22:03

     So we're showing, um-

121. 22:04

     Can you pair this vibe with something?

122. 22:06

     We're showing Wikipedia in this case. Um, oh, this is another fun, uh, skill that somebody in the team also developed, was the mood music skill, which calls a web service to compose music based on a single image and plays it.

123. 22:20

     Um, so yeah, it's able to kind of, [piano music] uh, compose some lo-fi music to go with the person's breakfast sort of use case. But it's a new, um, like, I guess, kind of paradigm in how we're able to like really, really, really easily extend the models.

124. 22:34

     And to do that in a pretty kind of low-code way, as you'll, you'll also see in a bit when we see, uh, when we dig into how this, uh, works under the hood.

125. 22:44

     Okay. So yeah. So examples here, and I'm not going to play all of these videos, uh, maybe to save time because I think you've seen many of them. So one is we can augment the knowledge base.

126. 22:54

     There's one pattern that we find interesting. We can produce kind of rich interactive content like flashcards for visualizations and I... Yeah. Um, uh, where we can... instead of like if you ask to summarize something in three bullet points, you can have like a JavaScript skill to kind of show those bullet points as a card instead.

127. 23:15

     Um, so you can just say like, "Summarize and show a card." Like, uh, "Summarize this," and if it thinks the, the card display will be more helpful for the user, that will get used.

128. 23:26

     Um, our music synthesis. Okay. So I'm going to skip forward a little bit from the demos to

129. 23:38

     how we build them. Okay. So what's actually happening

130. 23:42

     is, um, the way we've built the skills is they're efficient. Uh, so the instructions can be kind of loaded on demand. So this kind of uses a principle that you may have seen, um, uh, that you may have seen elsewhere of, um- Kind of, uh, well, um, like kind of progressive disclosure of conditional or conditional depth.

131. 24:06

     So instead of like in an MCP workflow where you need to describe everything about all of the functions, uh, that you need, the way we've structured the skills is there's a kind of one-line description.

132. 24:15

     The one-line descriptions of the skills is what the agent sees, and then if it thinks that sounds interesting, then it asks for more. It asks to load the skill.

133. 24:23

     It is a... Like, we teach it a skill to load a skill. Uh, and then it goes in and loads the skill, and it finds out all of the, the details about how to use it, what function calls it can use to, to do that skill.

134. 24:35

     So this is... Like, that pattern is particularly important for kind of token efficiency and frankly, like reliability on edge models. 'Cause if we had to load all of the details for all of the skills into the edge model, that would be a lot of context for the model to reason over.

135. 24:50

     And in a lighter weight model like that will, you know, that will hurt kind of performance ultimately, right? Because the lighter weight models are, you know, um, you know, they're really great to be able to run on device.

136. 25:03

     But in terms of reasoning ability over very, very long context windows, if you can, if you can, uh, have a more condensed context window, that will kind of, uh, up your batting average in terms of kind of quality metrics you're looking at to kind of ship a particular app.

137. 25:17

     Um, the second part is tool access. It helps us, um, uh, integrate new tools dynamically. So things like there's a ca- Like, we think of them in terms of there's a set of like input tools, like how do you get more information, which could be like Wikipedia or looking up a weather service or something like this.

138. 25:35

     There's a set of, um, uh... Then there's a set of, uh, things, uh, that are helpful to kind of present new outputs to the user, like kind of showing something on maps or showing something via cards.

139. 25:48

     So you can have skills both to extend the input and extend the output, you know, the output patterns that a model can do. Um, and it also kind of helps us to bring in domain-specific knowledge bases.

140. 25:59

     So that skill that's kind of calling Wikipedia, you could easily imagine that calling some, you know, customer CRM internally or some, um, asking data from a local rack system as well.

141. 26:12

     Okay. So our... Yeah, so our... Like, within our structure of skills, we have skill.md, and then there's optionally kind of scripts or assets where, um, the skills.md is the metadata that we always, uh, process.

142. 26:28

     Um, so, um, like, yeah, the example here is showing extracting text and tables for PDF, and we trigger on PDF for extraction. And then the instructions are only loaded when the model thinks that it requires that skill.

143. 26:41

     And yeah, this pattern is particularly important. Um, then in, um, within scripts, we do also support, um, uh, people kind of writing custom JavaScripts that can then be rendered within the app, because that's a really, uh, easy way to extend within, um, within both kind of iOS and Android systems.

144. 27:03

     Okay. So dig a bit deeper then [laughs] is we have within our system prompt, we have our own system prompt that we ship with the app. Then there's a set of skill descriptions which get added to the system prompt.

145. 27:15

     Uh, and then when the user asks for something, the model decides to trigger the skill by reading its metadata. Um, it then calls the load skill app that we have under the hood.

146. 27:25

     Um, it-- the tool response from that function call is then the contents of the skills.md file, uh, which is then in the context window. So it now knows about these functions.

147. 27:35

     Um, and then it calls the run... Oh yeah, so then that's, that also contains some JavaScript, and then we call the run JavaScript, um, uh, tool, which runs the JavaScript that we picked up from the, uh, from the skill file.

148. 27:52

     And then we call a response on that as well. Um, one other thing to kind of mention as part of this workflow is the... One of the things we did when we were porting Function Gemma for reliability is within the runtime, we have constrained decoding that applies, but it's kind of choose to only apply to the output

149. 28:12

     when we're generating a... when we're calling a tool. Um, and we can also... The constrained decoding can constrain it just to the particular tool that the, um, uh, that you're supposed to be calling.

150. 28:27

     So like in this system, instead of just kind of having generic JSON constraints, we know that there's a finite set of tools that the model is supposed to be able to use, so we can therefore have stronger constrained decoding.

151. 28:37

     This also kind of helps us have a more reliable system overall by using stronger constrained decoding. Um, we find that's less, yeah, for... We find this helps a lot for like the two billion parameter model.

152. 28:48

     As, as models get more capable, we find the, like the margin you get from this sort of strict constrained decoding, as we call it, is, um, it's less essential when you're running a very large model.

153. 28:59

     Like, if you were running a ten billion or a, a very large model. But for really small models on device, this is a very helpful tool that helps us, um, have stronger guardrails for the model, uh, so that we can up the quality so we have something that is, um, useful in production.

154. 29:18

     So then we support... I wonder. Wait. We'll see if this will play for us. We then support, um, within the app, we support, uh, you can toggle the skills you want to use.

155. 29:31

     Um, so even the skill descriptions, you can decide how many skill descriptions you want to have live at any given point in time. Um, so this is loading a piano playing scale, virtual piano, which, uh, will do its things.

156. 29:44

     You can actually tap the keys and play sounds if we can. [piano playing] Yeah. This is just more JavaScript stuff. But then, um, we can also... you can also load custom skills.

157. 29:56

     So you can write a skill yourself and load it from a URL, which is kind of fun if you want to experiment with kind of prototyping a skill with Gemma for an app idea you have.

158. 30:05

     This is something that you do. Oh, back there also, the skills can have like an API key as well. So like a secret key. So if you need to use a web service, that's something that the...

159. 30:14

     you can prompt the user to put in. This is showing then the mood music example again. But, um, we...

160. 30:22

     And we have a, on our GitHub page, we have a GitHub discussion where users are posting skills that they have written themselves. Um, and the skills in the community that we like, we are then able to kind of pull up and have as, like, featured skills in the app.

161. 30:37

     So if something developed so that, uh, ones that people develop, we can have a way of kind of showcasing useful community skills to the wider community as well. So this is add third party skill.

162. 30:50

     I think this is gonna show us. Okay. This is adding an animal intro skill. This is, like, a kids thing.

163. 30:59

     Uh, it's a lot. It's a lot. So, um, to the app.

164. 31:07

     Awesome. Yeah. Um, but yeah, the key point here is there's a really, really low barrier to extend the model, right? Um, and to extend the model in a way that is, you know, kind of relevant to downstream app users.

165. 31:21

     Um, so this is very, very easy. And we're gonna go a little deeper now into see just how easy that is.

166. 31:30

     If the clicker is... It's, uh, it's a good thing.

167. 31:39

     Okay. So yeah. So skill... Yeah. So skill architecture, when we go one step deeper, um, this is kind of relevant. So we have our own kind of orchestrator. We- it has a skill registry, and we call the load skill skill in order to load the skills.

168. 31:55

     And within skills then there's like a JavaScript skill. There's native intents. So at least within, um... For the Android system, we're able to call, like, kind of Android system intents or native intents, and you can c- you can just call those in your skill.

169. 32:10

     So things like if you wanted to turn off... on and off the Wi-Fi or something like this. So intents that are exposed to all Android that are available in JavaScript, you can certainly use.

170. 32:20

     Um, uh, then there's, like, kind of the role play, the skill.md, which is the persona and scenario data. And then there's the ex- specific skill and resources. And like I was saying before, this can include, um, this can just be JavaScript that runs entirely locally, which is a fully offline experience.

171. 32:38

     Or like in the music composer one, we actually kind of called a web API, and that needed an API key that then the user was prompted, and that also works within, uh, Gallery.

172. 32:49

     So, so then, yeah. Then the predefined tools we have under the hood, just to give a mental model of how it works, is we've got a load skill, which helps us load skills, run JavaScript, or run intent.

173. 32:59

     Um, and just the orchestrator just using these three skills is able to make the overall system work.

174. 33:08

     So this is another vi- video.

175. 33:10

     Now, let's test our effective TV and multilingual-

176. 33:13

     This is from our Gemini product manager showing, uh, Olivier.

177. 33:17

     Hey, Olivier. « Est-ce que tu peux trouver un restaurant français à San Francisco? » Please reply to me in English.

178. 33:25

     This is a restaurant roulette skill. This... Yeah. We actually have about, like... It was so easy to develop skills using, um, um, using both, um, like, Antigravity, right, or Gemini CLI.

179. 33:42

     We actually had the team do, like, 80 of these skills. So we had, like, lots and lots of things to, to do, uh, with lots of options in terms of kind of what to, what to show.

180. 33:50

     Or it was a lot of fun for folks to develop these. So here we can go in and just look at the, the structure itself, uh, of restaurant roulette.

181. 34:02

     Um, um, yeah. This is the under the hood part. So this required, uh, requireSecret true. Okay. We wanted to get an API key, and it co- searches for 10 restaurants, um, and then returns in...

182. 34:18

     Yeah. Okay. Returns the location and its cuisine.

183. 34:22

     Ah, and this is the index.j- dot JS file then, which is, um, where we can have a, um, yeah, simple kind of web view then kind of rendered to have the kind of roulette wheel.

184. 34:37

     And you can, like, obviously go and code all of this yourself, right? And there's full source code here for the examples. Cloud Code, we... Yeah, never got that actually published.

185. 34:48

     Um, the... But it works really well in Cloud Code as well. And we have both a source code for the example and a skill spec for getting started. But what,

186. 34:57

     uh... And there's full instructions on GitHub if you wanna try writing your own skill.

187. 35:03

     Uh, but this was, this was the pattern we actually used most. It was using, like, skills to write skills. So using something like, uh, Gemini CLI or Cloud Code or Antigravity.

188. 35:15

     Uh, this was our kind of favorite pattern, where we just say, "Hey, I want to write a skill for AI Edge Gallery." Um, this, this, this example works in Gemini CLI, um, where we just say, "Hey, this is the documentation.

189. 35:26

     Here are some examples of some skills that you can go and read." Then this is the skill I want to build, which was a offline arc- archiver. Um, oh yeah, the idea is when you...

190. 35:37

     Yeah, you can, uh, well, just based on things, pictures you took around London, for example, it would then be able to give you, like, there's that cool Churchill statue I kind of walked past.

191. 35:48

     It could kind of then... It would kind of go and research a bit about Churchill for you. And then on your flight back, you could go back into this skill, and it would have a bunch of information about the stuff that you saw that you could kind of read, uh, on the flight home, for example.

192. 36:01

     I think that was the inspiration for this one. Um,

193. 36:05

     uh, yeah. So it kind of... It fetches Wikipedia content and then stores it locally and then has a index. Um, also here with CLI, we can also even ask the, um- Uh, because, uh, we have an ADB skill in Gemini CLI, you can also ask it to test the skill itself by saying, "Hey, you have access to

194. 36:26

     a phone connected via ADB." And then Gemini CLI uses its Android ADB skill to go in and test that the app actually kind of works and, and that the skill is doing what it says in the app.

195. 36:36

     And, and we can just add it, ask it then. You can put this in a prompt and ask it to iterate and, um, it'll do some basic validation itself and return it.

196. 36:47

     Uh, yeah, so this, I think is going to be... If this plays, this will be an example of doing that. I could also have done this.

197. 36:56

     Um, yeah. So this is, um, this is us using Gemini CLI just to do that. And this works really robustly. I think of the maybe 80 skills our team did internally, maybe m- more than half were, were kind of vibe coded skills, at least initially.

198. 37:13

     Uh, which then really, really reduces the barrier for folks to kind of extend an LLM to do kind of new things in a way that's kind of useful, uh, for, for their audience.

199. 37:26

     Um. Take over the screen. Yeah. Yeah, go for it.

200. 37:36

     Uh, all examples here are running on the smallest model in the in-app, or is it in on device?

201. 37:42

     The-- We-- All of the examples here are defaulting to the, to the... Good question. Uh, so the question for the rec- recording is, uh, are all of the examples running on the smallest model?

202. 37:52

     In this case, the examples are running on the 4B model, right? Uh, the 2B model, the skills will work with the 2B model as well, and you can try that, but it's, um, yeah, your, your kind of mileage may, may vary, like simpler skills, fewer skills, that sort of thing, right?

203. 38:06

     But all of the examples you've seen are running on the 4B model.

204. 38:11

     Have you tried it with the Chrome CPP?

205. 38:15

     Uh, trying doing a version of skills in Chrome?

206. 38:18

     Yeah, so like using the Chrome CPP to, with the skill to actually use a local model to work in your browser.

207. 38:26

     Uh, we haven't, no. Yeah. It's a interesting thing to try.

208. 38:34

     Okay. Oh, yeah, and this then is quick shout out to this link. Uh, so a few things about Gallery that I should also mention are... Just quick check on the time.

209. 38:44

     A few things about Gallery that I should also mention are, one is Gallery is an open source project, and the open source project itself builds on top of kind of the LiteOr TLM kind of tooling that you saw kind of earlier as part of the intro.

210. 38:59

     So the app itself, you know, as well as being something that's kind of fun to use, so you can like, I, I guess kind of prototype kind of app ideas or to see, wow, is this possible in the model on a phone, or how fast would this be if I ran it on a phone?

211. 39:13

     Or is this sort of skill actually viable to do? As well as being able to kind of do kind of model prototyping, you also have kind of full access to the source code here and the, um,

212. 39:25

     and the models... The model also enables... Or sorry. And it also builds on top of the s- the same infrastructure like the open source LiteRT and LiteRT-LM, uh, acceleration infrastructure under the hood.

213. 39:37

     So if you see something you like in Gallery, you know, yes, you can deploy the Gemma model some other way as well. But if you want to get the same experience or the same speed, you can then just take, um, the underlying kind of open source APIs and, uh, pull a model from our Hugging Face page that

214. 39:54

     we'll look at more in a little bit and, uh, just run that directly as well. Um, so then as, as part of that, we also have a set of discussions on the Gallery, um, on Gallery that shows the, um, some of the community, uh, skills that have been uploaded from things from like, uh, cat entertainment to, uh,

215. 40:18

     um, blackjack, right? [laughs] I guess. Um, so these skills you could also, um, you can... People have posted these skills. So if, if you develop a skill, feel free to post it here and, uh, it'll make it more discoverable by other people in the community.

216. 40:32

     And then the ones here that seem compelling or maybe useful to a lot of people, we can add to our kind of third party preferred skills list in the app so that then it'll make it easier for folks to discover.

217. 40:43

     Um, yeah. So this is... Yeah, this is something we saw a reasonable amount of action over the weekend on, and it's been something that has only been up since last, last Thursday.

218. 40:56

     Okay, um, quick time check. Okay. So next up, I wanted to switch gears. So everything we saw with skills kind of really applies to, to, like from your question, applies to the 2B and 4B model, which mostly on mobile phones will be destined for like, um, uh, like system gen AIs where that'll really turn up in production.

219. 41:19

     Certainly for IoT or for desktop or, um, you know, um, for edge applications, you could just load that model yourself and run these sort of skill use cases in production on like that kind of Qualcomm IoT platform we saw earlier, for example.

220. 41:34

     Um, but to deploy, to deploy models in-app today and kind of ship them in production apps, like we're seeing more people are using smaller models to do that. Um, uh, so that we would call tiny models, which are less than one billion parameter models.

221. 41:49

     In our, in our view, these are the sort of things that we work with teams to deploy LLMs within their app, and these are the types of things we're seeing.

222. 41:57

     So I wanna just kind of briefly take a look at what that workflow looks like. Um, so LiteRT-LM, this is the engine that kind of, that powers, um, Gallery, right?

223. 42:09

     Um, and itself is a open source project that has, um, C++ and Java APIs. There's Swift APIs coming soon. It also has a Python API as of last week, which is kind of relevant for, um- Relevant for IoT developers who kind of like to work in Python.

224. 42:26

     Um, um, and that takes a li- an LLM file and has the required components to kind of fully run, uh, an autoregressive loop and expose that via easy-to-use APIs.

225. 42:41

     Yeah, so it's cross-platform, uh, C++ APIs, and where relevant, the same API can, uh, be used with hardware acceleration, like the Qualcomm example. Um, um, we also, for other smaller models in the past, we've published, um...

226. 42:59

     We don't have this yet published for Gemma 4, but in the past, we've also published models that work on, uh, MediaTek silicon as well and Intel silicon as well.

227. 43:08

     Um, so yeah, so over the course of the year, um, and as there are more and more smaller models available, we'll see broader, uh, NPU support.

228. 43:18

     So then the workflow is, you know, um, you start from Transformers. We use a package called LiteRT Torch that can, um, do, uh... That has some hydrogenative optimizations that will optimize for LiteRT and also has quantization built into the workflow.

229. 43:34

     Then that gives you, uh, a LiteRT-LM file, um, and you can then deploy that. Um, you can either prototype it with the Gallery app or just deploy it directly, um, to your kind of production candidate use case using LiteRT-LM in whichever platform, uh, you wanna work with.

230. 43:52

     Um, yeah, this is another view of that flow. I guess, yeah, for really advanced use cases, we have LiteRT Torch generative API, so if you actually want to write your own...

231. 44:03

     If you want to write your own Tiny model from scratch and train it from scratch, we support that workflow as well as just kind of fine-tuning stock standard workflows, and there's an API called the Torch Generative API that allows you to, um...

232. 44:17

     Basically, it's got kind of building blocks, uh, for LLMs, um, that supports many common LLMs that are, um, in native PyTorch that are in a way that give really, really good performance when you run them on-device.

233. 44:29

     Um, and are... Yeah, so that's the flow. Wow, this is, this is really in the weeds now. So we, [laughs] um, this is our stack to kind of deploy on NPUs.

234. 44:40

     Um, the... Wow. Takeaways from this slide are maybe two things. One is under the hood, we actually invest an awful lot of work in, um, optimization libraries. So we've XNNPACK, which is a CPU optimization library, and ML Drift, which is an optimization, optimization libraries for GPUs.

235. 44:58

     So we s- we have teams who work in these to ensure that both of these have excellent performance. Lots of Google's 1P apps rely on both of these libraries.

236. 45:06

     Um, so we're very motivated to ensure their, um, they have really great performance and work in the widest possible set of devices. And these are used what we call, like, the JIT workflow, like a...

237. 45:17

     where we produce a single artifact called a LiteRT-LM file or a LiteRT file that can work across CPU and GPU and get deployed to lots of types of devices.

238. 45:27

     For NPU, we need something a little more specialized. For NPU, we need to call a vendor compiler plugin upfront, and that uses an ahead of time compiler workflow. So you need to produce an artifact that's particular to a particular NPU.

239. 45:39

     Um, and then within our runtime, we, um, we call a dispatch to a particular, um, API that's allowed to dispatch work to the device driver of the NPU. Uh, but, like, both of these are available through a consistent API.

240. 45:56

     So even though the, the path, the JIT versus AOT affects the kind of the build workflow, the actual, the app development workflow is very similar, um, across, uh, NPU or CPU or GPU.

241. 46:09

     Export and inference is ac- really simple. Uh, we've LiteRT Torch to just export. Um, I know I've talked a lot about, uh, Gemma models because it's kind of like the Gemma week for us.

242. 46:21

     Uh, so we're very excited about Gemma at the moment. It's worth kind of calling out, we do support third-party models as well. So there's a Qwen 6B parameter model is what's there.

243. 46:30

     And those Qwen models also work in the app, right? Which is what you're gonna see on the right-hand side. And the app also supports just loading any LiteRT-LM file and kind of running it and getting benchmark stats, which we'll hopefully see in the video in a second.

244. 46:44

     Um, so you can run it like, say, on GPU, and then... Dun, dun, dun. Dun, dun, dun.

245. 46:50

     The latest Pixel that you're simulating here?

246. 46:55

     Wow. Actually, that's a great question. I... To be honest, I don't know. This... We do a lot of testing on Pixel and a lot of testing on, on S25 as well, right?

247. 47:09

     Um, so it... I, I assume it's one of those two devices, [laughs] right? Uh, but I, I don't know definitively. Um, and we... Actually, we didn't show it there because we...

248. 47:19

     Like, recently, like in the last release, we've added a, an optional icon underneath each chat, and if you click on it, it shows you the kind of prefill decodes, um, stats for the model.

249. 47:29

     So you can just... If you want, like, so you can, you can find any model we have on Hugging Face and, uh, load that in the app and then run to see the benchmark stats for this model on a particular phone if you want to.

250. 47:42

     Um, so yeah. So you can, um... And also in the command line, like if you run... If you're running on desktop, you can also, um, just use kind of LiteRT-LM run if you want to kind of do some desktop prototyping to kind of understand how models work in desktop.

251. 47:58

     You can also do that. And then, yeah. So I guess this is another... Yeah, this is another third-party model. This is FastVLM. This is a model from, um, yeah This is a model from Apple.

252. 48:11

     It's actually really, uh... It's a really nice, uh, VLM model that's only 500 million parameters. Um, and literally this is running with hardware acceleration on Qualcomm, uh, which is why it's running so fast.

253. 48:24

     Um, and this is running... Yeah, this is running on an S25, I think, um, 'cause it's Q- it's Qualcomm silicon. And this is literally we just have it running in a loop saying, "Describe the scene.

254. 48:35

     Describe the scene," uh, with video input. And it runs... Yeah, it runs really, really fast. Um, so this is a good example of what's possible with a model that's feasible to deploy on device.

255. 48:47

     Like, we haven't, we haven't, in this case, done 4-bit quantization, but I assume had we been motivated enough, we could have done that. And [laughs] we would then have, like, something that has, like, just...

256. 48:58

     That only requires an extra maybe 250 or 260 megabytes in your app to give this type of experience. So yeah, they're certainly feasible. And this is earlier when I was saying about models.

257. 49:09

     Like, this is an example of s- of a general purpose 500 million parameter model. Like, it's been trained on a fairly narrow set of things, which are, like, kind of scene descriptions or just, like, image to description.

258. 49:21

     So it's... Like, it's a narrow-ish use case, but still general purpose also, right? So this, this is a good example of the, the larger tiny models. [laughs] Uh, the smaller ones, like, that we have with, like, Gemma 3 270M, for example, uh, those models typically require fine-tuning to do a particular task.

259. 49:41

     Uh, but, um, yeah. But this is a good ex- This is why I included it. This is a good example of a general purpose, um, uh, tiny model that is, is very useful.

260. 49:51

     Uh, there's also... Like, it's worth noting, like, I, I don't think I've included in the talk, other examples of general purpose tiny models include there's a lot of s- very small transcription models, uh, or some narrow pairwise translation models out there at the moment as well, many of which we support, many of which are on our Hugging

261. 50:09

     Face page as well if you wanna kinda check them out. There's a, um, yeah. There's a bunch of, uh, things available on Hugging.

262. 50:17

     Oh, yeah. So that's... Yeah. Next slide is... This is just showing a handful of models that I, um, wanted to talk about. So function Gemma is one that we published with, um, a partnership with GDM last year.

263. 50:31

     This is function Gemma that was a general purpose model that you can further fine-tune for function calling, and there is Colab notebooks out there if you wanna look it up in terms of how to format data sets and how to, how to fine-tune f- how to fine-tune function demo for function calling.

264. 50:46

     The next two ones above that are Mobile Actions and Tiny Gardens. Mobile Actions is a... This is the example I mentioned earlier that has the 10 different mobile actions that are...

265. 50:55

     we fine-tuned ourselves, where we achieved, I, I think it was, like, 86 or 70% reliability on those. Uh, Tiny Garden is another. Like, this is a game, uh, that is...

266. 51:07

     we built inside in the Gallery app that you can play with as well. It's just another example of a voice-to-function calling fo- fine-tune model, uh, that you can, can use.

267. 51:17

     Um-

268. 51:17

     What is the IT, uh, suffix indicate?

269. 51:21

     Um, IT is instruction fine-tune. So we, in models, uh, when GDM publish models, they sometimes have PT suffixes and sometimes have IT suffixes. Um, PT is when, um, PT is when we publish a model just after pre-training before fine-tuning.

270. 51:40

     That's kind of helpful for kind of expert users because you can do all of your own instruction fine-tuning, so you can fully control the model's personality, and you're not trying to, like, unlearn some fine-tuning we did.

271. 51:53

     In this case, the way we teach the model how to do function calling is actually via instruction fine-tuning itself. So when we publish a model for further fine-tuning for function calling, it already has a, like, function calling personality, and that's actually what we wanted to publish.

272. 52:09

     Um, so in this case, this model is for further fine-tuning, but it's an IT model. More typically, if you see for larger models, um, you... Like, certainly for the Gemma 3 family, we had both IT and PT checkpoints.

273. 52:24

     Uh, yeah. We had both IT and PT checkpoints for those depending on if you had a large volume of data and you want to do full fine-tuning yourself. Like, I think there's, there's a separate conversation on, like, sovereign AI use cases that, uh, s- some of the DeepMind people are gonna do later in the week.

274. 52:39

     And I, I imagine that will be a, "Hey, start with our pre-trained checkpoint, and now add a huge corpus of data, of sovereign data or enterprise data, and you can fully fine-tune a 27 billion parameter model to do, um, stuff."

275. 52:52

     Um, yeah. Oh, another example here is embedding Gemma. Um, I know that's not a... Technically it's not an LLM, but embedding Gemma does, um... That's a text embedding model that, um, that, uh, we published with the Gemma team last September, and, uh, that does text embeddings for RAG-type use cases.

276. 53:14

     But that's also, like, a very, very strong embedding model that only takes 300 million parameters. It's another example of a kind of high utility, um, uh... Yeah, I, I guess it's not technically an LLM even if it is a transformer inside.

277. 53:27

     But, uh, it's another example of a tiny model that's, uh, very useful for on-device use cases.

278. 53:34

     Okay. Yeah. Uh, I've kind of already covered this, I think, 'cause we have, um, yeah, both AOT is... AOT compilation is our workflow for hardware acceleration. On device is best for distributing small models to lots of platforms.

279. 53:50

     Um, okay. Yeah. And then I kind of covered this as well. So LiteRT-LM itself is built on LiteRT, and it supports, like, all of these types of models, and you can find, like, non-LLM models also on Hugging Face.

280. 54:05

     This is kind of relevant because you typically, when you're building a more complex app, you need some things around an LLM, like a voice activity detection model or a denoising model.

281. 54:16

     Um, so many... These models are also available using just the LiteRT runtime. Like, LiteRT-LM is the runtime that has the full auto regressive loop it builds in LiteRT. And there's a set of LLM models that are available to use with LiteRT-LM, but there's lots of supporting, um, uh, just kind of, um- Uh, how do you say?

282. 54:36

     Um, non-autoregressive models that are available to use with the LiteRT API that are kind of relevant to deploy, uh, they're typically used to deploy a full app.

283. 54:47

     Okay. And this was the earlier thing I was saying about, um, for advanced usage, you can fully customize models. So if you wanted to write your own, uh, Llama or Moonshine or Phi or Qwen variant, uh, you can using the code in this directory.

284. 55:06

     Okay. I'm gonna have probably on track to finish a little early, I would say. Okay. Um, so that's it in terms of the tiny models, examples, and workflows. Um, so there's a lot and there will be...

285. 55:22

     Yeah. So there's, uh, a lot we can do now with very tiny models. Um, 500M class are available for some standard features, uh, for smaller LLMs. We've had a lot of success fine-tuning them, uh, for apps.

286. 55:36

     And what I'm going to show next is an example of a app that we've built using, uh, tiny LLMs. Uh, so this is an app that's available on iOS, uh, only for some reason, um, uh, called AI Edge Eloquent.

287. 55:52

     Um, what this is, is a kind of transcription, um, model. But instead you may have noticed that as a speaker, I say lots of ums and um, right? If you got the transcript of this presentation, it was, uh, it wouldn't be a great transcript.

288. 56:06

     There would be lots of, uh, lots of, uh, interjections. This, uh, Eloquent was built for that type of transcription user story where you, um, where it does dictation, but then as a separate, um, kind of, uh, automatic kind of polish, uh, step where it can kind of remove all of the kind of interjection and, and words.

289. 56:29

     So if you wanna dictate a message for use later, um, it's able to, you know, clean up idioms of speech. One of the other things it has as well is a, uh, a biasing, uh, list.

290. 56:43

     So one of the things we found ourselves is like if we're talking about LLMs, you would say, "Oh, have you trained a LoRA for this or got it?" And any standard, um, any standard transcription service would when you say LoRA, it would translate that to the name L-A-U-R-A is typically what happens.

291. 57:00

     Whereas you can give it a list of kind of keywords or technical terms as well, and then the model will kind of, um, uh, bias to those words, right?

292. 57:08

     Um, because that's how real people spell. So you can-- So the, the kinda key features here is, A, it runs entirely offline, B, it's got its own kind of biasing dictionary, uh, if you wanna call it that, and then, C, it kind of cleans up the text if you see, uh, uh, in the middle, it has this

293. 57:24

     kind of polish step, right? Um, and overall, yeah, this is the kind of gives a pretty, uh, neat offline, um, uh, yeah, a kind of no, no cost, right?

294. 57:36

     Like, it goes, goes back to that kind of cost motivation at the start as well, 'cause there are some paid services that do this. Um, but it can give you really clean, uh, cleaned up text.

295. 57:44

     Is, is that last step done by changing, like, se-setting special tokens in the RL? Or like, is it done last, like, by overlaying a map or something?

296. 57:53

     Which is this? The-

297. 57:54

     The, the last step where, like, the LoRA example.

298. 57:57

     Is it done-

299. 57:58

     We-

300. 57:59

     ... by the second model, or is that done in the model?

301. 58:01

     Ish is... We're gonna... Give me, give me two slides. I'll answer your question, but yeah, good question, right? The question for the roo- for the recording was, is the, is the polishing done inside in the main model or outside, and how is biasing applied?

302. 58:16

     And we're gonna answer it in two slides, hopefully.

303. 58:19

     Yeah, personalization. Yeah, this is the, uh, personalization. So you can add, um, you can add things like LoRA, [laughs] the example we always use 'cause it's kind of, uh, near and dear to our hearts.

304. 58:33

     So it allows you to... If you want to connect it to your Google account, it can import stuff from Gmail or something, um, and look for unusual words, and then add them to their biasing list, or you can add them in yourself.

305. 58:44

     The sort of things people usually put in here are names because, like, um, models will get uncommon names wrong frequently, right? Um, as well as technical terms, right? So like Jianing is and Cyril, these are two of the people on the team who developed the app.

306. 59:00

     Uh, yeah, so no surprise, uh, that's the example we have.

307. 59:07

     Okay. So this is where we... Yeah. So the text polishing engine is what we're calling it in this stage. [laughs] Uh, so we have two steps here. Uh, important to note that both of these, um...

308. 59:23

     Yeah. First-firstly, I'll describe, then I'll talk about models. So microphone input goes into a speech recognition engine that delivers a unfiltered transcription. Then in the bottom half, we have the kind of personalization flow where you get a set of kind of uncommon or unique words that then goes into relative terms.

309. 59:39

     Both of these go into a text polishing engine, which is then a, um, dedicated L- mini LLM just for text polishing, right? We could probably have built like one LLM to do all of this, right?

310. 59:51

     But it's actually... Like, sometimes, like one of the other realities of mobile development is sometimes it's, like with tiny models as well, there's this kind of like modularity story.

311. 1:00:01

     So that same transcription engine in your app, you may have some other use case for that, and you may not want to pay, um, you know, you can recoup the cost of those weights by using them in multiple places, right?

312. 1:00:12

     So this, this is a pattern we see emerging as we build more of these types of apps is that kind of modularity playbook. So yeah, in theory, those two models could be stitched together.

313. 1:00:21

     In practice, it's kind of a more pragmatic choice to have separate models.

314. 1:00:25

     Again, this is more adept as well.

315. 1:00:27

     Yeah.

316. 1:00:27

     Now you can-

317. 1:00:28

     And you can also inspect what's happening in the middle. It's easier to debug, right? Um, we see the same a little bit with voice-to-function calling, like should, you know, uh, as well, but that's another story for another day is, um...

318. 1:00:38

     But it's, uh, yeah. So- Okay. Then the other thing to point out, the ASR engine and text generation, these have little... I don't know if you recognize the Gemma logo in the corner.

319. 1:00:51

     These aren't, like, officially published Gemma models, but these are-- We've basically done the same workflow that we're advising to other people to do, to do. We have taken the Gemma models, the smaller Gemma model, right?

320. 1:01:03

     This would be a derivative of like... This would be something from the Gemma 3 270M lineage, is the best way I would describe it, right? And we've basically taken that model and done a fine-tuning just to have, um, uh, just to have a transcription engine and then just to have a text polishing engine.

321. 1:01:23

     Uh, we've fine-tuned them. Generally, what that workflow looks like, to give you a bit more insight, is we'll use a workflow of using a much stronger LLM in the cloud to generate lots and lots of synthetic data that corresponds to the type of thing we want.

322. 1:01:36

     And then once you have a few, you know, low digit millions or tens of millions, depending on how ambitious you want to be of synthetic data, you kind of put that into a fine-tuning workflow, and you fine-tune the base tiny model you're working with to get a derived model.

323. 1:01:51

     And, you know, that same workflow we're showing here, um, you know, uh, you can then use to... Like, we've used that internally to kind of ship a stronger, um, to ship a stronger, uh, note-taking app, right?

324. 1:02:05

     But that same workflow is, uh, the reason I'm showing it here is, like, a real life example that like, um, of how we're using Gemma-derived tiny models in order to be able to, like, build new production apps.

325. 1:02:15

     And we're using this flow in, yeah, in-- for, for lots of other use cases internally as well, supporting Google 1P products in lots of other ways that, um, those products will probably talk about themselves in time, right?

326. 1:02:29

     Um, but yeah, like, like, smaller Gemma models are really, really powerful for this type of use case. Um, and we're seeing good mileage coming from this. Yep.

327. 1:02:38

     Are you gonna, like-

328. 1:02:39

     Yep

329. 1:02:39

     ... combine it with, like, keyboard? Because this is a very useful thing.

330. 1:02:42

     Yeah.

331. 1:02:43

     Yeah.

332. 1:02:44

     Yeah. So this is-- that's what the text polishing engine does. Like, as part of that we, um... Wow. Like, the text polishing engine was able-- was trained, was instruction fine-tuned, I guess, to kind of have a system prompt of like, "Hey, these are your special words.

333. 1:03:01

     Uh, please, uh, please correct, uh, anything that sounds like these words to these words, and then also remove, um, interjections or lack of clarity, or even things like, 'I forgot to say this,' or, 'Scratch that.'" Like things like this.

334. 1:03:17

     Probably a scale, huh?

335. 1:03:18

     Yeah. No, this-- in this case, it's not a scale because this is a tiny model. We've actually just trained the behavior into the model. It's true with a, with a four billion parameter model.

336. 1:03:25

     Maybe there's a version of this you could do as a scale, which would actually be an interesting side project. Uh, yeah. Maybe there is a version of this you could use as a scale.

337. 1:03:32

     But for the tiny models, the playbook we're generally seeing is really useful is synthetic data, data generation with a larger model, and then pick an off-the-shelf like Gemma 3 270M, right?

338. 1:03:45

     Or similar, and run with that with fine-tuning. And then you can, um,

339. 1:03:52

     combined with quantization, you can then ship a pretty compelling narrow feature to a very wide set of users, uh, powered by an LLM that works on lots of devices.

340. 1:04:01

     Yeah.

341. 1:04:01

     Do you have, um, like in GitHub repo or like somewhere workflow how to fine-tune it? Because you don't just fine-tune it, right? You had like warm up the training and then start.

342. 1:04:12

     Yeah. We do. With the Gemma 3 270M publication, we do have a Colab notebook. And with Gemma 3 270M and Function Gemma, both of these, when we ship those models, have Colab notebooks that show how to do kind of fully fine-tune, um, models.

343. 1:04:27

     Uh, how to, how to do full fine-tuning. Yeah.

344. 1:04:30

     Uh, okay. So regarding the fine-tuning.

345. 1:04:33

     Yeah.

346. 1:04:33

     What's your experience? Uh, let's take the, the function calling example.

347. 1:04:38

     Yeah.

348. 1:04:39

     Uh, and you said you had, uh, eighty percent chance of hitting the ten, ten functions. And-

349. 1:04:45

     So it was eight... Uh, we finished that-

350. 1:04:47

     That, that is probably after fine-tuning.

351. 1:04:49

     Yeah.

352. 1:04:50

     And could you say some numbers before? What, what's about-

353. 1:04:53

     Oh, wow

354. 1:04:53

     ... are we talking?

355. 1:04:54

     Oh, wow. You're-

356. 1:04:55

     Are we... Is it ten percent? Is it fifteen?

357. 1:04:57

     Like forty something percent. Forty something percent to eighty-six percent, right? Within that eighty-six percent as well, right? 'Cause I don't wanna, um, there was one... Oh, wow. We had, we had ten functions, and maybe two of those functions dragged our average down a lot.

358. 1:05:12

     It was like eight simple functions that were over ninety, like ninety-three percent type thing. So we, on very simple functions, we had like really, really high reliability. And then the, yeah, I, I just forget the actual details, but there was like two that kind of brought the average down to more like kinda eighty-six percent, which was where

359. 1:05:28

     we finished. Um, there's also a blog post you can read about that for more detail. Yeah. Um, but we'll see this in... Yeah. We, we see this with smaller models all the time.

360. 1:05:39

     It's like a, like our experience is on a given eval, um, like fine-tuning is between twenty and forty points on the eval. So it's like it's a really, really significant win for tiny models, right?

361. 1:05:52

     Like when we're talking two hundred million parameters, right? Um, or two seventy, right? Like we published with 3M. Yeah. Um, their fine-tuning is, is essential for most things, right?

362. 1:06:05

     Unless you have a, unless you have a model that is already published, you know, to do a narrow task, right? Like you'll find a transcription, you'll find transcription models out there that are in that size that work really well at one task and don't require further fine-tuning.

363. 1:06:20

     But that's because they've been scoped to the narrow task from the outset, right? But if you want to do your own narrow task, yeah, right? And ship something to lots and lots of devices, then yeah, fine-tuning is, um, at least for now, it's, it's the workflow of choice.

364. 1:06:34

     Yeah. It may change.

365. 1:06:36

     Do we get a Colab that can use the web view?

366. 1:06:40

     Because now with these models it's possible.

367. 1:06:43

     Um, that's a good question. So we have a collab to get as far as the LiteRT file, right? Um, our support for LiteRT-LM on web is work in progress, right?

368. 1:06:59

     Like we, um, yeah. Yeah, is-- I'll just say that. Like please look at our GitHub. Please look there for latest status, right? You, you'll see it.

369. 1:07:08

     One more thing.

370. 1:07:08

     Um, yeah.

371. 1:07:08

     So like will we get a fine-tuning manual for Gemma 4 by any chance?

372. 1:07:13

     Um, there are... So Gemma 4, what was announced last week was small models and medium-sized models, right? In the past for Gemma 3, we also published tiny models, right?

373. 1:07:26

     So, um, uh, but this is like they were the first models we shipped for Gemma 4 were last week, right? So there will be more Gemma models in future, I imagine, right?

374. 1:07:37

     Um, there are some fine-tuning already available for the larger models, right? But for the tiny models we've published at this point in time, right, and if anybody's reading or listening to this talk in a few months' time, please search the web for the latest information.

375. 1:07:51

     But as of right now, the tiny models we have published are Gemma 3 tiny models. Um, and they do have for function Gemma and for Gemma 3 270M, there are fine-tuning workflows, uh, available, right?

376. 1:08:02

     For Gemma 4, as available last week, there are some workflows, um, there are fine-tuning recipes in, uh, Vertex and elsewhere, I believe for those. And but, um, check out the other Gemma talks, uh, later in the week and you'll, you'll see and hear more.

377. 1:08:20

     Uh, oh yeah. So the Eloquent thing is actually available on iOS if anybody wants to give that a go. And that-

378. 1:08:25

     Yeah. Not in Europe, though.

379. 1:08:27

     Not in Europe?

380. 1:08:27

     Yeah, I could not download it.

381. 1:08:28

     Oh, okay.

382. 1:08:30

     Yeah. You can find it on web browser, but not on the-

383. 1:08:33

     Oh

384. 1:08:34

     ... App Store.

385. 1:08:35

     Wow.

386. 1:08:35

     So that needs to be enabled, I guess.

387. 1:08:37

     That's really helpful feedback.

388. 1:08:39

     Yeah.

389. 1:08:39

     I'll pass that along. Okay. Uh, yeah. And then wrap up. Yeah. The kind of key takeaways. System GenAI, medium-sized models in, in our, uh, or small models in our parlance, right?

390. 1:08:51

     Um, will be kind of turning up in a, a mobile device near you. Those same models are excellent for use in embedded systems, embedded platforms. Um, uh, at least for now, given the memory we have in mobile phones, which doesn't look to be getting larger anytime soon given the cost, um, tiny models are kind of where it's

391. 1:09:10

     at, um, for deploying things widely, right? We hope to make those easier and easier. Like we hope to have stronger and stronger models available, uh, through the partnership with GDM, and we want to also make the, the fine-tuning workflows as easy as we can, right?

392. 1:09:24

     To enable, to make this kind of more accessible. Um, but yeah, happy just to kind of share what we've been doing in both of these fronts. So I think we've nine minutes if anybody else wants to ask any questions.

393. 1:09:36

     Happy to... Yeah.

394. 1:09:38

     Is it challenging to handle sort of safety on the edge models? 'Cause you don't have the ability-- you don't have like a hosted server where you can do all of this checking.

395. 1:09:46

     You just have to shovel all of this into training.

396. 1:09:48

     Yeah.

397. 1:09:49

     You've got to restrict things to a-

398. 1:09:50

     Okay

399. 1:09:50

     ... certain device.

400. 1:09:51

     Yeah. Like great question. So question was about safety on edge models. So firstly, the Gemma team spend an awful lot of time on this, and I would defer you to them for all the questions on safety for the models that were published last week.

401. 1:10:04

     But, um, they do... Like it's really top of mind. Like I, I... Like they spend an awful lot of time on safety for those models. Um, additionally, for like for system GenAI within AI Core, for example, right?

402. 1:10:18

     What actually ships there, and you'll probably hear more of this from other people, is within system GenAI, when that actually ships as part of the OS, not as a role model that we published last week.

403. 1:10:27

     The system vendor will typically have some sort of input and output safety checker on the model, right? Uh, as a kind of aftermarket addition because th- there's particular things for their product they, they want and they don't want, right?

404. 1:10:39

     Um, so that's another layer. For smaller models then, for like really tiny models, the... Like safety there is really important, but generally the way, like at least for us, the way we've deployed tiny models is to within a very narrow kind of API or task.

405. 1:10:56

     Like you can imagine with Eloquent, like the app you saw, that kind of the risk profile there, like technically that's more like of a regenerative app than a generative app, if you know what I mean.

406. 1:11:06

     It's not going to fully make things up on the fly. So, uh, we tr- Like, yeah. So there's a, um, you know, you can kind of look at the scope of what the model is trying to do and the API surface and the functional surface.

407. 1:11:18

     And typically tiny models have, to make a tiny model work, it generally has narrower functional scope, which allows for a more kind of like... You still need to do safety, but it's a narrower problem you need to solve when you're kind of looking at it, right?

408. 1:11:31

     Uh, is what I would say. Yeah.

409. 1:11:34

     Um, is there a way, like, um, place where I can look for how to deploy small, like little bit bigger Gemma models on like 5090? Because I have a 5090.

410. 1:11:44

     Want to deploy it on that and then try it out. Is there where I can look it up?

411. 1:11:49

     Ooh. So the... Okay. The... Hmm. So on 5090. [laughs] So I believe, I'm not sure if we've... Like I... So I know we don't have specific documentation for that, but our tool flow does support, uh, NVIDIA GPUs.

412. 1:12:08

     Mm-hmm.

413. 1:12:08

     NVIDIA is also, uh, like a Gemma partner, right? They, um, they have supported some of the Gemma launches in the past. So it may be, it may be if you look on NVIDIA's own webpage, you may see some of that through their TensorRT-LM.

414. 1:12:21

     Um, I know they've supported some of the Gemma launches in the past. But we don't... Like I can't point you to specific documentation as a fast answer. But they're the two places I would check.

415. 1:12:31

     Okay.

416. 1:12:32

     Yeah. Okay. Yeah.

417. 1:12:32

     Yeah. Sorry.

418. 1:12:32

     Yeah, that's all right.

419. 1:12:33

     So the examples that you showed for like the agent skills there, obviously like, uh, one skill execution, like based on the intent. You run the, you run the skill that's appropriate-

420. 1:12:42

     Yeah

421. 1:12:42

     ... where that's a single skill, the-

422. 1:12:44

     Yeah

423. 1:12:45

     Sort of like change the architecture for like multi-skill execution. Or is that difficult with smaller models?

424. 1:12:52

     It's a question of ... So we do actually support multi-skill execution, right? Um, it's something ... So within the app, if you download the app, you can actually define, um, you can define, um, the skills you want loaded, right?

425. 1:13:08

     And you can just toggle them on or, on or off, right? Um, and then you can easily say, like if you're very, very specific in your prompt, which is like, wow, um, like, uh, if you're very specific in your prompt, which is like, you know, look up this topic in Wikipedia, summarize the three bullets, and then the

426. 1:13:26

     play as- display as flashcards. If you're really, really specific, that's going to work, right? Um,

427. 1:13:32

     frankly, like we've only had this model a couple of weeks. So we're still putting miles on the clock, uh, and seeing what are the boundaries of how, like, how much can we do skill stacking and skill chaining, right?

428. 1:13:44

     Uh, so we're ... Yeah, we're literally still in the mode of putting miles on the clock there. So most of the examples we were publishing were single skill. But even in the, even in the diarization app, right?

429. 1:13:54

     If you recall there, um, like Alice, the person doing that, she asked for, um, she asked to, oh, summarize my mood or what time am I meeting up with Amy?

430. 1:14:05

     And then, oh, Wikipedia is going to come up. Ask for blah, blah, blah. So like within ... In that example, she was able to kind of show within a single conversation individual turns using individual skills.

431. 1:14:17

     Um, uh, yeah. But s- like skill stacking within an individual prompt, I believe we've seen that work where you're pretty explicit, but also we're still ... Like frankly, we're still learning, right?

432. 1:14:27

     The boundaries of, you know, like this, uh ... Yeah, we're still learning the boundaries of what's possible with these classes of models.

433. 1:14:34

     Sorry, a follow-on question.

434. 1:14:36

     Yeah.

435. 1:14:36

     So, um, so it's like the decision to identify which skills are important for the prompt. And given that it's a small model, do you pro- do you like do instruction tuning from a larger model to build that intention?

436. 1:14:47

     Nope.

437. 1:14:48

     Or is it-

438. 1:14:48

     Nope

439. 1:14:48

     ... really good at-

440. 1:14:49

     This is, um ... Like the model was ... Wow. The model was changed to be really good at agentic workflows and really good at function calling, right? It wasn't trained ...

441. 1:14:59

     Like, nothing in the Gemma model was trained specifically for our skill pattern, right? Uh, that just kind of came afterwards. Like, when we got the model and we started playing with it, it was like, "Wow, this works."

442. 1:15:08

     And then, "Oh, can we do this," right? So it was more that kind of workflow. Uh, so there's no specific training for our skill structure, right? But they ...

443. 1:15:17

     You know, the team spent lots of time doing general purpose thinking and function calling, right? Um, like the GDM team did lots of great work there to give us some ...

444. 1:15:26

     A general purpose model is really strong. But there's nothing special for our app.

445. 1:15:30

     Okay.

446. 1:15:30

     Right? So if you had a, if you had a slightly different take on skills, and maybe there's a better skill architecture than the one we've shown you today, right?

447. 1:15:37

     It's entirely possible. Um, yeah, you, you could expect to be pretty successful, right? There's no, there's no like hidden secret sauce in what we've shown.

448. 1:15:46

     Thank you.

449. 1:15:46

     Yeah. Uh, I'll take class ... Um, sorry. I think you're maybe next.

450. 1:15:51

     Yeah, so-

451. 1:15:51

     Yeah.

452. 1:15:51

     I'm building like an agentic app.

453. 1:15:53

     Yeah.

454. 1:15:53

     I've been using all the kind of model providers, and I tried to switch to the Gemma models.

455. 1:15:58

     Mm-hmm.

456. 1:15:58

     I've got a few challenges. The first big challenge is context window. How ... You know, particularly once you start doing the agentic loop-

457. 1:16:05

     Mm-hmm

458. 1:16:05

     ... the, you know, the context-

459. 1:16:06

     Yeah

460. 1:16:07

     ... um, go up to like 100K quite easily.

461. 1:16:09

     Yeah.

462. 1:16:09

     Can you talk a little about context window on, on the, on, on the-

463. 1:16:12

     Yeah

464. 1:16:12

     ... E2b particularly, before-

465. 1:16:13

     Ooh, E2b.

466. 1:16:15

     Yeah.

467. 1:16:15

     Okay. Um, yeah. I would, I would defer you to the Gemma team for official guidance. Like the, the medium-sized models have a context window of 128K. And the smaller models I would actually need to double-check, right?

468. 1:16:30

     I think you said 32K.

469. 1:16:31

     32K. Yeah. That's why I was wondering if it was 32K. So like our implementation, like in Gallery, we default to like 8K or 12K or something just for performance reasons, right?

470. 1:16:41

     But the models do support up to ... Like E2b and 4B support up to then 32K, and the other models support up to 128K.

471. 1:16:48

     With 32K, you use a lot of memory, right? Is that right?

472. 1:16:54

     Um, wow, I wonder if we have stats for that in the model card. It's ... For the E2b and the E4b model, the, the memory footprint for a larger context is, um ...

473. 1:17:05

     It's, it's not as bad as you think, right? [laughs] Uh, there's actually ... Put a lot of optimize it. Like the team optimized that metric for those models because it was targeted for edge use cases.

474. 1:17:16

     So the amount of kind of KV cache that's required for each input token, that was something that was optimized. So the models behave pretty well on that front. I don't have, I don't have a number off the top of my head of like bytes per input token, um, uh, to give you, but it's, it's, it's, uh, it's

475. 1:17:32

     good for its model class is what I would say. Yeah.

476. 1:17:34

     Okay. And now the second question. Do you have the iOS version of the, um, Edge, uh-

477. 1:17:40

     Yeah. The, uh, AI Edge Gallery works on both iOS and Android.

478. 1:17:43

     But in terms of being open source. Because I think the Edge, Edge version for-

479. 1:17:47

     Ooh. On, on macOS. Uh, that's a good point. I will put that, uh, in the coming soon bucket, right? Uh, it's certainly one of the items on our to-do list.

480. 1:17:58

     Because we published ... Yeah, the iOS app we only published for the first time in January. Um, whereas the Android app has been available since last summer, right? Uh, but we do ...

481. 1:18:09

     Like our intention is to have, um, a what you see is what you get, uh, experience for developers, right? So you can use the app, have fun, experiment with the models, and then also get the source code and see how it's built, et cetera, right?

482. 1:18:23

     Um, so yeah, that's cer- certainly our intention.

483. 1:18:28

     Yeah.

484. 1:18:28

     But is there a trade-

485. 1:18:29

     Probably last question because we're at-

486. 1:18:30

     Is, is there a trade-off-

487. 1:18:31

     Yeah

488. 1:18:31

     ... between, uh, like fine-tuning individual models for a specific task and then like actual amount of memory on device they consume? So it's like, you know, you've got your quantized model will be like 3.6 gigabytes or something.

489. 1:18:44

     Yeah.

490. 1:18:44

     Does that not like add up over time when you're like chaining the models?

491. 1:18:49

     Ooh. So to clarif- so for which, for which model is your question? The- I've just downloaded whatever the E2B- The E2B model. Yeah ... so if you've got multiple models fine-tuned- Ah, okay ...

492. 1:19:03

     presumably So f- yeah. So for, for E2B, we would kind of recommend customization via skills or via prompting, not via fine-tuning, right? Um, for the smaller ... So for the small models that are published, we recommend customization through skills and prompting, right?

493. 1:19:19

     Um, for tiny models, um, we would recommend, uh, we would recommend customization through fine-tuning. If you were deploying a smaller model, right, I don't ... Like, on, on ...

494. 1:19:32

     The other path that is available to you for the medium, for the small models is LoRA fine-tuning, right? So I know Apple supports that in their foundation model framework.

495. 1:19:40

     Uh, you can check with the AI Core speaker if that's on their roadmap, right? There's an AI Core speaker who has an AMA, uh, coming up. You can check with him about their roadmap for this.

496. 1:19:50

     But certainly if you were deploying that on an embedded system, right, like, uh, if somebody asked me about deploying the 2B on an, uh, on a robotics platform, I would be like, "Absolutely, you should fine-tune LoRAs for each of your things, uh, for each of your tasks."

497. 1:20:03

     And then, like, our runtime supports loading the model and, like, hot swapping LoRAs, so you don't even need to kind of load and unload the model to load and unload, uh, LoRA adapters, uh, that is built for that particular use case for, like, robotics or IoT platforms.

498. 1:20:17

     Um, and that's ... And those LoRAs then are, like, maybe

499. 1:20:22

     ... Yeah, it depends on the radix you choose, but they're much smaller. It's like,

500. 1:20:27

     you know, maybe 16 to 100 megabytes depend- or actually even smaller, like eight to 100 megabytes, in that kind of range depending on the, the radix you use. Yeah.

501. 1:20:36

     Thanks so much. Yeah. Cool. All right. That's a wrap. I'm gonna let everybody get lunch. Yeah. Thank you. [clapping] [outro music]