Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
0:11:31It's so good to be here. Thank you for
0:11:33being with me this morning. First
0:11:35session of the day. I see everyone here
0:11:37is a morning person. So I'm here to talk
0:11:39to you about what will you build when AI
0:11:42is 10 times faster. We're all builders
0:11:44here. So we're all thinking about what's
0:11:46next.
0:11:48Just a bit about me. Um my name is
0:11:50Natalie. I'm a co-founder of Gimlet
0:11:52Labs. At Gimlet, we're building an
0:11:54inference cloud that's designed to
0:11:55produce much, much faster tokens at high
0:11:58efficiency, leveraging different
0:12:00categories and types of chips. At
0:12:02Gimlet, I work on our kernel generation
0:12:04side of things across different hardware
0:12:06types. Um, another thing about me is I
0:12:10love agents. I use my coding agents for
0:12:12everything, but I really, really hate
0:12:14waiting.
0:12:17And one thing that given that kind of
0:12:19surprises me is when you're building an
0:12:20ultra fast inference cloud, people ask
0:12:22me this all the time actually. They're
0:12:24like, isn't inference already fast
0:12:26enough?
0:12:28And I'm like, what?
0:12:30But they have a point, right? Because I
0:12:32can do things and we can all do things
0:12:34with our agents so much faster than we
0:12:36could have done before. For example, uh
0:12:38this slide here, I actually made pretty
0:12:41using my agent and it took me with my
0:12:44agent much less time than it would take
0:12:46me to do that by myself. I'm not very
0:12:48good at design. And so I think that from
0:12:52an endto-end perspective, obviously
0:12:54we're getting more done, but in terms of
0:12:56is it fast enough, like that's still a
0:12:58question. So let's take a look at what
0:13:00happened when I used my agent to make
0:13:02that slide look pretty. You can see the
0:13:04ugly before slide on the right.
0:13:08So, we're going to press play here.
0:13:12And I said to my agent, can you improve
0:13:14the visual look of this slide? I love
0:13:16Figma. I use Figma for everything. So,
0:13:18of course, I'm going to use it with my
0:13:20agents. So, what we can see on the
0:13:22bottom here is the timeline of what
0:13:25happened. And it the text is a little
0:13:26small to see, but red is inference. Red
0:13:28is the time spent that the model is in
0:13:30inference. Green is the time it's spent
0:13:32making tool calls. and blue is the time
0:13:34that it's spent taking my input. And we
0:13:38can see that, you know, this this
0:13:39agent's doing good work. It's doing a
0:13:41lot of thinking, but I'm waiting here.
0:13:44I'm actually waiting so long I sped up
0:13:45this video 9x because I didn't want all
0:13:47of you to have to wait for the full turn
0:13:49to end. It ended up being about 2
0:13:52minutes long. But we can see at the end
0:13:54of this after my one input, we have this
0:13:56nice pretty slide at the end. So, you
0:13:58know, on one hand, this is actually much
0:13:59faster than a human can do. But on the
0:14:02other hand, in those about two minutes,
0:14:03I have probably moved on to another
0:14:05task. I'm not really focused on this
0:14:07anymore. I've context switched.
0:14:11And when we look to, you know, the
0:14:13inevitable progression of inference
0:14:16getting faster and faster, I think it
0:14:17helps to look back in time and look back
0:14:20to the internet era. Um, I mean, we're
0:14:22still in the internet, but like you know
0:14:23what people canonically call the web
0:14:24era. And what we always see with a new
0:14:27shift is that with each new major wave,
0:14:30new capabilities are unlocked. And so we
0:14:32do things that weren't possible before
0:14:34that technology, like using an agent to
0:14:36edit my slide. So I think we should take
0:14:38a step back to 2000 in the early days of
0:14:40the internet. I think many people here
0:14:42weren't even alive, so it'll be a
0:14:43history lesson for you. And we're going
0:14:45to look at a little application called
0:14:48Squirrel Mail. Squirrel Mail is actually
0:14:50open source, so you can run it today. It
0:14:52was a web application for uh you know
0:14:53doing your email back then.
0:14:56So on the left what we have here is I'm
0:14:58running squirrel mail and I'm running it
0:15:01with simulated 2000 dialup speed and
0:15:05then on the right it'll run with 2026
0:15:07speed. So the speed that we're used to
0:15:09today. So the both sides are going to
0:15:11take the same actions but um the left
0:15:14side I'm using it with that simulated
0:15:16lower uh speed internet
0:15:21and we can see that you know it's doing
0:15:23its job. Back in 2000 loading an image
0:15:27like this would have blown people's
0:15:28minds but our modern eyes are like wow
0:15:31that took forever to show up.
0:15:33And so we can see these lags. we can see
0:15:35a noticeable thing, but on the same
0:15:37time, someone might have looked at this
0:15:38back in 2000 and said, "This is fast
0:15:40enough. I don't actually need this to be
0:15:42faster."
0:15:46The thing is though, all of the major
0:15:48giants back in that era did find out it
0:15:51mattered. Pretty much every major
0:15:53company from Amazon to Yahoo to Google
0:15:56to Bing, all of or being at part of
0:15:59Microsoft like all of them found that
0:16:01even very small delays cost them users.
0:16:03It cost them people coming back. It cost
0:16:06them people doing as many clicks. It
0:16:08cost them literal money. Even a hundred
0:16:11milliseconds of delay added on to the
0:16:13typical experience had a demonstrable
0:16:16impact on the bottom line. And even when
0:16:19you got rid of that delay, those same
0:16:20users didn't come back to the platform
0:16:23as often as they did without that delay.
0:16:29And this makes sense because fast
0:16:31products keep us in flow state. They
0:16:33provide a stickier UX. About 100
0:16:36milliseconds feels instant to us. One
0:16:38seconds, one second of weight, I still
0:16:41keep my train of thought. It's not a big
0:16:42deal. I don't really notice it. 10
0:16:44seconds, we're going to start turning to
0:16:46something else from a psychology
0:16:48perspective. Maybe we pull out our
0:16:49phone, maybe switch to another agent.
0:16:51But today, from data from June, the
0:16:53median coding agent turn is 63 seconds.
0:16:59Each workload has different speed
0:17:01requirements. We don't need the same
0:17:03speed for everything. But over time, our
0:17:05expectations do rise in the fastest days
0:17:08of fastest internet in the squirrel mail
0:17:10days would feel incredibly slow to us
0:17:12and we can keep we can expect that trend
0:17:14to continue for inference speed as well.
0:17:18There's a problem though and the problem
0:17:19is everyone knows that we have massive
0:17:22compute scarcity. Today's infrastructure
0:17:24is under incredible pressure to deliver
0:17:26two things at the same time. More tokens
0:17:29and faster tokens. Unfortunately, these
0:17:32goals are fundamentally at odds because
0:17:35faster tokens require more energy to
0:17:37produce than slower tokens. So, with the
0:17:40same energy footprint, you can't make as
0:17:41many of them.
0:17:44Back before this was a session and not a
0:17:47keynote, thanks to upgrading this to a
0:17:49keynote everyone. Um, we uh we're going
0:17:52to play a game that I made uh called
0:17:54Token Mint Factory. And you can actually
0:17:57play this. I will not be offended if you
0:17:58play it right now even, but just so you
0:17:59know, it does not work on phones. It
0:18:01only works on your laptop. And basically
0:18:03in this game, you're running a token
0:18:04mint factory. You're printing literal
0:18:06tokens. And there's different modes. The
0:18:08modes are you can play to maximize how
0:18:11quickly you can put out one single
0:18:12token. Or you can maximize how many
0:18:15tokens you can put out in 60 seconds. Or
0:18:18you can play the SLA mode where you have
0:18:20to maintain a 20 second SLA on each
0:18:22token for it to count, but you're still
0:18:23trying to put out as many as you can.
0:18:25And as you play, you'll quickly see that
0:18:27the strategy is extremely different for
0:18:29doing fast tokens versus doing slow
0:18:31tokens. And neither one is actually
0:18:33better or worse from a decision
0:18:35perspective. It's just a matter of what
0:18:37the application needs and what makes
0:18:38sense for your factory.
0:18:41So I took real runs from people at my
0:18:44company that played this game and I made
0:18:46a paro frontier of these tokens in the
0:18:48game. And we can see there's a real life
0:18:51paro frontier of this game as well. If
0:18:54we have faster tokens, we don't have as
0:18:55many. And if we have more tokens,
0:18:57they're not as fast. But there is a way
0:19:00to get more up and to the right. And up
0:19:01and to the right is where we want to go.
0:19:03We want more faster tokens. I don't want
0:19:05to compromise. I want both. So the game
0:19:08has a concept of adding upgrades to your
0:19:10equipment. And what we can see is that
0:19:12compared to the no upgrade baseline,
0:19:14these technological innovations help
0:19:17drive us up into the right and get more
0:19:19tokens and faster tokens.
0:19:23So how do we upgrade our real life
0:19:25assembly lines so to speak to make
0:19:28making our actual tokens faster? Uh one
0:19:31thing that we do at Gimlet for this is
0:19:33we really believe that you have to use
0:19:35the right accelerator for the right job.
0:19:37And so we actually take models and we
0:19:40split them up into different pieces and
0:19:42we run each piece on a different
0:19:44category of accelerator. So we might run
0:19:46one piece of the workload on a GPU and
0:19:49another piece of the workload on
0:19:51something like an SRM ccentric
0:19:52accelerator. And we find that even with
0:19:54the network overhead, this delivers
0:19:56dramatically faster tokens. It's
0:19:57essentially like the same thing as the
0:19:59upgrade to the equipment in the token
0:20:01factory when you better match the
0:20:03accelerator to the job. Unfortunately,
0:20:05we have ended the time of free lunch
0:20:07transistor scaling, right? So now we
0:20:09have to look to other approaches to make
0:20:11things faster.
0:20:14And so with this, it's a little hard to
0:20:16read, but we can see the same real life
0:20:18paro frontier for a frontier model. If
0:20:21you look at, you know, simple
0:20:23homogeneous GPU inference, it's very
0:20:25high throughput. And then the latency
0:20:26drops off a little bit. You don't you
0:20:28can't get it as fast because the y-axis
0:20:30is how many tokens that I can make and
0:20:32the x-axis how fast these tokens are for
0:20:34each user. And we can see that depending
0:20:37on how you split the model up because
0:20:39there's different ways of doing it. You
0:20:41can get much faster tokens for the same
0:20:43power footprint.
0:20:45You can get many more tokens for the
0:20:48same speed or you can get more and
0:20:51faster tokens. So as an industry we need
0:20:54to continue pushing pushing pushing on
0:20:56this because otherwise our poor data
0:20:57centers are just going to keel over.
0:21:01So I just wanted to show a quick demo of
0:21:03this. So um using a coding agent build
0:21:06the same app using heterogeneous
0:21:08inference that leverages different chip
0:21:09types versus the traditional one.
0:21:14So this is going to be the game snake.
0:21:16And we can see on the left the
0:21:17heterogeneous fast one's done. I can see
0:21:20in my flow state I can keep adding
0:21:21things. I can keep playing but on the
0:21:23right side it's still going. And by the
0:21:25way I still think the left side's too
0:21:27slow. We need to make that even faster.
0:21:31And then we can add a competitor to the
0:21:33game and we can see even with the tool
0:21:35calls, even with all the things that
0:21:36it's doing, it's going to complete and
0:21:39then the other one will still be hanging
0:21:40because it's fundamentally about how can
0:21:42you drive more efficiency and better
0:21:44performance with the same resources.
0:21:49It's going to go on. I'm not going to
0:21:50make you guys wait for this.
0:21:53One thing that people often point out
0:21:54too is is inference the bottleneck? Our
0:21:57tool calls the bottleneck. And I think
0:21:59this is a great point. If we look back
0:22:01to the turn timeline from the time that
0:22:04I was spending updating my slide, we can
0:22:07see that 80% of it was inference and 18%
0:22:09of it was tool calls. If we simulate
0:22:12that same exact thing and have 10x
0:22:15faster inference, we can see that 67% of
0:22:18that will be tool calls at that point if
0:22:20we don't optimize the tool calls. That's
0:22:23why it's important that we have to
0:22:24constantly attack the bottlenecks of the
0:22:26application and get the lowhanging fruit
0:22:27where we can. Today, people are focused
0:22:29on inference speed and maybe we haven't
0:22:31optimized the tool call side and how
0:22:33tools are called as much because it's
0:22:35not the bottleneck. If I cut away the
0:22:37tool calls on the upper one, it wouldn't
0:22:38even really matter to the end
0:22:39application. But once we get much
0:22:41faster, we have to think about how to
0:22:43call these tools much more efficiently
0:22:45and much more performantly because
0:22:46otherwise they're going to be the reason
0:22:48that we're all waiting around.
0:22:53I think the good news is that optimizing
0:22:55these speeds and what we're all
0:22:56collectively doing as an industry is a
0:22:58verifiable task. If I know what my
0:23:00inputs and outputs should be, I can have
0:23:03agents do this very very full stack. And
0:23:06that's where we're increasingly moving
0:23:08to. We're having self-improvement loops
0:23:10with these agents to drive better
0:23:12efficiency on the fly as our inference
0:23:14is running. And at Gimlet, we're
0:23:16building recursive self-optimization for
0:23:18this that pulls from low-level telemetry
0:23:20from our data centers all the way up to
0:23:22the model layer and trying to use all of
0:23:24it to in to improve the overall
0:23:26efficiency of the workload because if
0:23:28you just optimize one small piece, it
0:23:30might not make that much of a dent.
0:23:35I want to show another demo because I
0:23:37think that cyber security is on
0:23:38everyone's mind right now. Um, we're all
0:23:41seeing like the huge incredible advances
0:23:43in AI driving awareness that there's all
0:23:46these vulnerabilities in our software
0:23:48and I think that this is an area where
0:23:50faster inference can really really help.
0:23:53So this is a dumb fake scenario but it's
0:23:55a real application and a real demo where
0:23:58imagine that you're running the dog club
0:24:00and that's a website for dog lovers to
0:24:02upload and vote on pictures of dogs.
0:24:05However, unfortunately, like we often
0:24:07see, one of your node dependencies has a
0:24:10vulnerability and this is a real
0:24:12vulnerability. This is really a node tar
0:24:15and you're running a security agent on
0:24:17your application and it's trying to
0:24:19monitor live traffic and see is there
0:24:21anything going wrong that I don't know
0:24:23about. This agent doesn't know about
0:24:24this vulnerability, but it does know its
0:24:26job is to protect your application.
0:24:31So, let's do a quick turn with normal
0:24:33user traffic. We're going to upload a
0:24:36dog biscuit.
0:24:39And on the upper side, we have the
0:24:41application logs. And on the bottom, we
0:24:43have the defender. And the defender is
0:24:44monitoring, but it's not in the critical
0:24:46path because it's not fast enough to
0:24:48actually respond to every user request
0:24:50live. It's monitoring things async.
0:24:54And we can see biscuit gets uploaded and
0:24:56we see that this is an okay, you know,
0:24:58this is an okay upload. The defender
0:25:00doesn't see any problem.
0:25:02But now what we're going to do is have
0:25:04this defender operating when there's a
0:25:05malicious upload. And this upload is
0:25:07exploiting the real vulnerability where
0:25:09in node tar there was a release where
0:25:11you could basically overwrite files that
0:25:13you weren't supposed to overwrite based
0:25:15on the destinations of what was being
0:25:17unzipped.
0:25:19So in this one we have Ranger, a fake
0:25:24bad uh malicious dog. Not really a dog.
0:25:27And what we're going to see is that the
0:25:30defender is going to figure out that
0:25:32this is a malicious thing. The agent is
0:25:34smart enough to detect that it's doing
0:25:36something that it shouldn't be doing,
0:25:37even though it hasn't seen this exact
0:25:38vulnerability before. But unfortunately,
0:25:41Cat's malicious upload has taken over.
0:25:44So your website's gone. It's too late.
0:25:47You didn't react in time.
0:25:52But if we retry this same exact
0:25:54malicious upload, but this time with an
0:25:56ultra fast defender, and again, this is
0:25:58a real a real demo fake use case. Um, we
0:26:01can see that with ultra fast inference,
0:26:03we can react to things and do things
0:26:05online much more quickly. And this
0:26:07should change the way that we think of
0:26:09how to use agents in the first place.
0:26:10We're all thinking about how to use them
0:26:12in an async way, but we should think
0:26:14about how to use them in online ways as
0:26:16well. So we can see on the bottom that
0:26:18the worker blocked their request and it
0:26:20was like what are you doing? This is
0:26:21obviously wrong.
0:26:26So faster internet took us from squirrel
0:26:29mail to ondemand highresolution
0:26:32streaming, right? Like if you just look
0:26:34from this to this, it's crazy. If you
0:26:36showed this to my like 5-year-old self
0:26:38or something, I would have been like my
0:26:39mind would have been blown. And this is
0:26:41just one of many things that I can
0:26:42select. So, like what does a 10,000
0:26:45token per second world look like? We're
0:26:47at in the like low hundreds today. What
0:26:50does it look like when we get to a
0:26:51million tokens per second? Like what
0:26:52does that unlock for us?
0:26:56And I think that we should all ask
0:26:58ourselves this too. It's important in AI
0:27:00to skate to where the puck is moving,
0:27:02right? It's changing so fast. We need to
0:27:04be ready. We need to think boldly. It's
0:27:06not enough to just do what we've always
0:27:08been doing. I think we've all had the
0:27:09experience where we make an audacious
0:27:11request to our agent and after the
0:27:13latest model release, it just works. And
0:27:16it's going to be the same thing with
0:27:17latency. And so I think that we can
0:27:19think about ideas that are both coming
0:27:21soon when we hit a certain latency
0:27:23threshold, what we can do, but also the
0:27:25stuff that is real life sci-fi. Like if
0:27:27you look at something like Jarvis or
0:27:29something, there's no way today's
0:27:31inference could be fast enough to power
0:27:32that experience for Iron Man. But we
0:27:34will get there. So, I'm really excited
0:27:37about online use cases, online security,
0:27:40online things of like I want to just
0:27:43tell my agent what I want to build and
0:27:45it's building it right in front of me
0:27:46and I don't even have to type anything.
0:27:48I think we're going to have new patterns
0:27:50of interaction. You know, think tab
0:27:52complete for everything. How can we have
0:27:54the user side of things be as quick and
0:27:56immediate as possible because the
0:27:58inference side is also just as fast to
0:28:00accommodate it. And we're going to have
0:28:02so many more agents. We're going to have
0:28:04even maybe transformers driving robots,
0:28:06right? There's so much that we can build
0:28:09when this gets unlocked. And I think as
0:28:11builders on the frontier here, we should
0:28:12all be thinking about this, too.
0:28:16Thank you so much for being with me um
0:28:19this morning. Uh you can find me on X at
0:28:21NSino. Um we're hiring at Gimlet. Would
0:28:24love to talk to you all. And then if you
0:28:26want to play the token mint game, uh you
0:28:28can also find that QR here. Thank you so
