Beyond Better Assistants: Software That Can Make a Decision

5:06:58>> Give me a sec to make sure this is

5:06:59working. Heck yeah. I'm Diego Almeida

5:07:03and this talk is I love you so jeving

5:07:06much. Um glass animals reference. We're

5:07:09not just weird. Um like I said, I'm

5:07:12Diego Almeida. I'm known as the Jev guy,

5:07:15sometimes Jevy Pinkman. Um, also

5:07:18co-created chat GPT, RLHF, and Instruct

5:07:21GPT, but I assume that for the crowd

5:07:23today, you all know who I am, at least

5:07:25mostly. Um, I have like a bunch of AI

5:07:30specific stuff I want to get to, but I

5:07:31like going into my favorite topics in

5:07:33AI. First one is what the is

5:07:36actually going on and where is all the

5:07:38automation. If you've heard me say this

5:07:40before, I'm sorry, but it's such an

5:07:42important thing to beat into the field

5:07:44because people still do not get it and I

5:07:46need to yell this until everyone really

5:07:48understands what the is going on.

5:07:51And if you're an AI, you should have a

5:07:52really good answer about why AI can do

5:07:55so many awesome things and yet still be

5:07:57so useless for a bunch of other ones. My

5:08:00relationship to chat GPT is somewhat

5:08:02unique. One of the few people who hates

5:08:04on chat GPT who made it. I don't hate

5:08:07chat GPT to be clear. I love using

5:08:10agents. I love using coding agents. I

5:08:12love chat bots. They're just quite

5:08:13limited. And in particular, I think it's

5:08:15a little tragic how much they've

5:08:17bifurcated the field into overpromise,

5:08:19underdeliver. Really, really terrible

5:08:21stuff for the world in my opinion. And I

5:08:24like to talk about like the two cults in

5:08:26AI and how we can resolve the cognitive

5:08:29dissonance that everyone I think should

5:08:32be having unless they have a very good

5:08:33answer. There's cult one, which is, you

5:08:36know, the cult of like EAs. They look at

5:08:38all the benchmarks and they say like not

5:08:40only are LM surpassing human level on

5:08:43basically every benchmark they have been

5:08:45for two years we've been announcing AGI

5:08:47three months away for two years now even

5:08:49since like the 03 days do people

5:08:51remember that no one even car cares

5:08:52about that model anymore and it was that

5:08:55that was like peak internal to openai

5:08:57AGI hype and on the other hand you look

5:08:59at the nonbenchmarks and you look at the

5:09:02reality and you see the rest of the

5:09:04world especially if you guys go outside

5:09:05and leave San Francisco a lot of the

5:09:07world is like really unchanged despite

5:09:10AI being so cool. And I think AI

5:09:12is transformative. It's a rare point of

5:09:14view to actually think AI is cool and

5:09:17transformative and also like

5:09:19underwhelming the world right now. And

5:09:21the tricky part is this is the same

5:09:23field and we are all seeing the same

5:09:25facts. So what gives you know why is AI

5:09:28jagged so to speak like why can it be so

5:09:30good at some things and so not good at

5:09:33other things and what is the aams razor

5:09:36explanation for that um I will be

5:09:38blasting through my slides fast I'm

5:09:40sorry maybe I could slow down for people

5:09:42taking pictures um but uh I'm noticing

5:09:45that uh this is going to go fast is all

5:09:47I'm saying so I think that we as a field

5:09:50should have an aams razor explanation

5:09:52for why are there some things that are

5:09:55Not just good, they're unbelievably

5:09:57good, superhuman, like millennium price

5:09:59problems in math. But on the other hand,

5:10:02they're not just bad, they're so

5:10:03useless, they can't even be used to

5:10:05touch accounting. That sounds way easier

5:10:07than million prize problems in math,

5:10:08right? I could do the latter. I

5:10:09definitely could not do the former. And

5:10:11I was a mathlete. So what gives here?

5:10:13What is the reason for this? And I

5:10:16normally love this as an interactive

5:10:17thing, but I will instead just tell you

5:10:19my answer. Sorry for the power dynamic.

5:10:21Um, everything on the left side is

5:10:24intrinsically a human in the loop task.

5:10:26Like deep research, I could not do

5:10:28better than deep research. The models

5:10:29are so good at that. But if gun

5:10:32to my head or gun to your head, you had

5:10:34to pick between me and deep research.

5:10:36And the answer had to be correct, not

5:10:38well written, you'd probably pick me

5:10:39every time. And that's the distinction

5:10:41between like optimizing for automation,

5:10:44which tends to be a lot easier and a lot

5:10:46simpler. Yet m AI keeps failing at this.

5:10:49Every year there's like an infamous

5:10:51failure of AI failing to even automate

5:10:53drive-throughs. And that should be kind

5:10:55of shocking to people. And my claim is

5:10:58actually for these tasks pleasing a

5:11:00human is nothing to do with doing the

5:11:02task. What you want to do is basically

5:11:04be an abstraction away. You want the

5:11:06decision to be made and you want to get

5:11:08back into doing automation and code is

5:11:10the language of automation. Therefore,

5:11:12you need to integrate with code. This is

5:11:13a lot but I will summarize it again. I

5:11:15get the irony that uh you know I've

5:11:18co-created RHF and I do believe RLHF is

5:11:20part of the problem and RHF in

5:11:23particular is why all AI today does

5:11:26require a human in the loop with an

5:11:28asterisk unless the task is like very

5:11:30trivially verifiable and can run on its

5:11:32own but still requires a human in the

5:11:33loop to verify it afterwards. It is

5:11:35because we literally put it in that is

5:11:37what the HF is in RHF. And my claim is

5:11:40that there's a better way. You know, my

5:11:43summary of the state is 100% of LMS

5:11:45today are optimized for assistance. You

5:11:47might talk about RLVR, but the RLVR

5:11:49models are very bad to interact with if

5:11:51they've not been RHFed. The result is

5:11:54LMS become too good to be true at

5:11:56assistance. They are just so

5:11:58good. Like, you should never pick me

5:11:59over chat GPT for your chat GPT queries.

5:12:02But for actually doing like you know run

5:12:05in the background cognitive work you

5:12:07should not trust LMS at all because they

5:12:09have not been optimized for that which

5:12:10is why they're so bad at the tasks of

5:12:12automation and I would also guess that

5:12:14automation is a lot more valuable than

5:12:16assistants because you can do way cooler

5:12:18deeper I'm going to get into this later

5:12:19but I think there's going to be so much

5:12:21stuff that we can do and we are just

5:12:22scratching the surface of that right now

5:12:24and that's the tragic thing about AI uh

5:12:26the state of AI for me let's talk about

5:12:30the evolution Um, I I can't tell if this

5:12:33is appropriate, but it's too late. Um,

5:12:36oh god, don't take pictures of this one.

5:12:38Um, cool. Oh, I'm getting a thumbs up.

5:12:42Yeah. Uh, excellent. So, we're we

5:12:44instead of doing RHF, we just have to do

5:12:46something else. We need to think from

5:12:47first principles. How do we optimize for

5:12:49machine native AI? You guys hopefully

5:12:51know the interface of Jev and that is

5:12:52what we are doing. more important than

5:12:54optimizing for its interface, we are uh

5:12:57rethinking how AI should be done because

5:12:59this is what we did again for Arlf for

5:13:02Sutton's bitter lesson. I'm going to

5:13:05loosely paraphrase to make my point

5:13:06easier to say is that scaling um and

5:13:10compute matters more than algorithms.

5:13:12This is why people become like scaling

5:13:13pill jironies. And I would argue that

5:13:16there's an even bitter lesson than that

5:13:18which is data always matters more than

5:13:20compute. We are learning that quite a

5:13:22bit in the field now, but it has been

5:13:23obvious since the beginning of time.

5:13:26I'll tell you when the this is the one

5:13:27you should be photographing. I would

5:13:29argue that the the bitterest lesson is

5:13:31doing the right task. And doing the

5:13:33right task is like a weird thing to

5:13:35think about especially for ML people

5:13:37because they like algorithms. They want

5:13:39number goes down or up or whichever is

5:13:41good direction. They don't want to think

5:13:42about the real world. But this is always

5:13:44how you make gigantic gains in

5:13:46performance. This is how the big

5:13:48breakthroughs happen. 100% of the time

5:13:49it comes from this and this is true for

5:13:52classical RL games. This is why Sutton

5:13:54had his bitter lesson it which is data

5:13:57is derived by compute when you're doing

5:13:59rollouts like there's there's why would

5:14:00you need off policy data when playing

5:14:02chess or something like that and doing

5:14:04the right task is defined by the rules.

5:14:06So actually when you're doing RL you

5:14:08don't even need to think about the right

5:14:09task for pre-training we actually as a

5:14:12field had to do a lot of thinking about

5:14:14the right task. I don't know if people

5:14:15remember the birth days with um masked

5:14:18language modeling, but it took a while

5:14:20and it was very unclear to figure out

5:14:22that next pro token prediction was super

5:14:24cool. Not because it was better

5:14:26at masked language modeling. You know,

5:14:28the objectives on their own aren't

5:14:30special. It's about what you can do with

5:14:31them afterwards. And it turns out next

5:14:33token prediction is incredible at

5:14:34generating strings, which is what these

5:14:36pre-trained models were so good at. So

5:14:38no level of mass language modeling even

5:14:40if you had today's compute and

5:14:41algorithms would make it as interesting

5:14:43as GPT3 was because it was just doing a

5:14:46different task and data you know the

5:14:48internet is obviously a giant

5:14:50bottleneck. Now we're like you know

5:14:51burning books or whatever in order to

5:14:52get more data but data obviously matters

5:14:54a lot more than compute and people keep

5:14:56like rediscovering this and figuring

5:14:58this out. It's extra true for making

5:15:00LLMs useful. um before RHF even existed

5:15:04the data for RHF didn't exist and you

5:15:06could like it was night and day I'll

5:15:09just show you the next slide this is the

5:15:10probably the relevant part this is what

5:15:11this was the work in the RHF paper and

5:15:14[snorts] the high level was that even

5:15:16with the dumbest algorithms in the world

5:15:18even with a GPT2 sized model you got

5:15:21something like GPT9 level performance

5:15:23like real GPT9 not today's like fake GPT

5:15:26numbers like the numbers used to mean

5:15:28like a 100x scaling and effective

5:15:30compute for pre-training and they've

5:15:31given up on that because it just can't

5:15:33scale anymore. And still even with that,

5:15:35it would take something like GPT7 to9 to

5:15:37match GPT to the model from like 2018 or

5:15:41whatever it was at the task of

5:15:43instruction following. That is kind of

5:15:45wild because you think big LMS are so

5:15:47important for this and they are, but

5:15:49that is not the most important thing

5:15:51compared to even the gigantic jump of

5:15:54doing the right task. And spoilers, this

5:15:56is what we did for Jeff, right? like we

5:15:57made this gigantic leap in the field

5:15:59that no one realized could be coming

5:16:01because we were optimizing for a task

5:16:03and very few people think about the

5:16:04right task. Um this is why Jev has the

5:16:09Jeb Nissiqua. Um that's a really smart

5:16:12bun I think. Um

5:16:16little bit of a pity laugh, but thank

5:16:17you. Um you get what you optimize for. I

5:16:20I'm looking for a better word than

5:16:21reliable. So if you guys have

5:16:23suggestions, let me know. But there's

5:16:25there's this reliability of intelligence

5:16:27that's really missing in LLMs. I often

5:16:29call it you get Einstein 95% of the

5:16:32time, but Mr. Bean 5% of the time. And

5:16:34if you can't sell which is which, how

5:16:36the are you going to automate

5:16:37anything? And instead, instead of like

5:16:39focusing on like sometimes occasional

5:16:41smarts that you can then verify, what if

5:16:43you focused on extreme extreme narrow

5:16:45focused reliability so you can get that

5:16:48smarts every time? And this doesn't show

5:16:50up on benchmarks. It is the reason why

5:16:51people love Jev. And this is also I

5:16:53realized in hindsight we obviously

5:16:54didn't prepare for a chat GPT moment for

5:16:57developers. But it happened because

5:16:59people in the AI space just overpromise

5:17:02underdel so much that having a

5:17:04model that actually don't do that was a

5:17:06surprise. And it is not the speed cost

5:17:08or even interface. This is something

5:17:10that you know my team tells me I

5:17:12shouldn't be telling people so that we

5:17:14can have a long-term advantage and I

5:17:15just care too much. Like it's reliable

5:17:17intelligence. Make sick models please.

5:17:19even if we, you know, if it doesn't

5:17:21involve our models. Um, we see that Jev

5:17:24as like a foundation for smart software.

5:17:26I'm going to skip all over this, but I

5:17:27think software is going to be so

5:17:29back. I think the narrative of SAS

5:17:31apocalypse is obviously not going to

5:17:33happen, but I think that even calling it

5:17:36a SAS Palooa is like an understatement

5:17:38and I will get into that a bit and you

5:17:40guys ideally are the perfect crowd for

5:17:41this. I'm so excited for this part

5:17:43because that's the part that's new to

5:17:44me. Um, I want to make the distinction

5:17:47between smart software and just in time

5:17:49software. The the crazy part is AI is so

5:17:52transformative and yet somehow despite

5:17:55AI being so awesome, software has

5:17:58essentially unchanged in the last decade

5:18:00except being cheaper to write. Isn't

5:18:02that weird? Isn't the it weird that the

5:18:04best we can do in software is having a

5:18:06chatbot to the side that sometimes

5:18:08interacts with a software but cannot

5:18:10take expensive decisions for you. That

5:18:12is kind of like a a weird thing if AI

5:18:14was really that transformative. And I

5:18:16think the problem is we're focusing too

5:18:17much on coding agents and not enough

5:18:20about putting the intelligence into the

5:18:21inner loop, which I'm going to get more

5:18:22into a bit. And people are jeving.

5:18:25There's been tons of successes. We don't

5:18:27need to flex this. There's so many cool

5:18:28things, but my AIE secret is going to be

5:18:32that the field isn't jeving hard enough.

5:18:34And this is what Swix said that I can

5:18:36make a call to action for you guys

5:18:37that's unique. And hopefully I have

5:18:38enough time for this. I think the world

5:18:40still has LLM Stockholm syndrome. We are

5:18:43still thinking from a really LLM pill

5:18:45place which is I think tragic you know

5:18:48like I I I love the software saying the

5:18:50tools you use change the way you think

5:18:52and I think that it's changed the way we

5:18:54think way too much and we're not really

5:18:56rethinking software like I think the

5:18:58field needs. I think that we could just

5:19:00do be doing so much more. Hopefully I

5:19:02can try to do my best for my remaining

5:19:03time to convince you guys of that. Um so

5:19:06let's Jev to the future. Um

5:19:10I an analogy I love bear with me is

5:19:13about LEDs. Before there was the blue

5:19:16LED there was red and green LEDs. These

5:19:18were really limited in capability and

5:19:20they were used for like simple

5:19:21indicators and you know like clocks and

5:19:23stuff like that. And when the blue LED

5:19:26was made discovered I actually don't

5:19:28really know how things outside of

5:19:29computer science work. Um, but when it

5:19:31existed, I guess, um, it enabled not

5:19:34just like 50% more use cases, which what

5:19:37you'd expect about like changing colors

5:19:38from two to three, it enabled so many

5:19:40more. Like now LEDs could be used for

5:19:42white light. It could be used for like

5:19:44the displays that we all use right now.

5:19:45And it created a revolution in lighting.

5:19:48And it's it's it's unfathomable how much

5:19:50bigger the capabilities of LEDs were.

5:19:53Just ex, you know, adding a new

5:19:55primitive that composes together with a

5:19:57old primitive. So you can see where I'm

5:19:59going here. maybe. And I think it's a

5:20:01similar story for software. Software is

5:20:04so incredible. Ignore AI for the

5:20:06last, you know, half decade. But

5:20:07software has done so many insanely cool

5:20:10things. We've done a lot of stuff, you

5:20:12know, like do I love using SAS? Not

5:20:16really. But does it solve problems? Abso

5:20:18lutely. And it's incredible how

5:20:20much we've done with three fundamental

5:20:22logic gates. You know, like we are doing

5:20:24this with like simple binary operations

5:20:26that can do things like greater than,

5:20:28less than, equal. It's kind of nuts how

5:20:30much software has been built from such

5:20:32simple primitives. And my claim is that

5:20:36what we are doing with AI now when we're

5:20:38making a intelligence machine native

5:20:40this is kind of conceptually adding a

5:20:42fourth semantic logic gate. What if we

5:20:45can program with common sense with

5:20:47intention with human understanding? Like

5:20:50do we still need reexes with like crazy

5:20:52things or can we just say like we want a

5:20:53Gmail email address or a real looking

5:20:55one? There's so much that we can do that

5:20:57we're going to need to push the f, you

5:20:58know, push the frontiers of cheap,

5:21:00smart, fast in order to get done. But I

5:21:03think that there's going to be like so

5:21:04many things here. And I think just like

5:21:06the blue LED, it's going to be way

5:21:08bigger than just increasing the

5:21:10representational ability of software by

5:21:1233%. Because it's adding effectively

5:21:14another kind of sort of logic gate. And

5:21:17this sci-fi is the stuff that I think

5:21:18will be impossible to predict. Just like

5:21:20who could predict how powerful the

5:21:22internet would have been because it

5:21:24requires it's like much more grassroots

5:21:25and decentralized and it requires people

5:21:27to be empowered to do insane stuff. Um I

5:21:30call this software research. Um that's

5:21:34not a joke. Um but uh I I think that

5:21:37there's going to be like an explosion of

5:21:39what software can do and we will want

5:21:42people to be exploring those boundaries

5:21:44of software. Not necessarily because

5:21:46it's valuable, but because they can.

5:21:48Because we have like new frontiers to

5:21:50explore and this is what the early

5:21:51internet was like. Should all of those

5:21:53websites existed? Probably not. But like

5:21:56it was very cool and we found a lot of

5:21:58awesome stuff from that. And I think we

5:22:00we, you know, we have another

5:22:01opportunity for that. So we should not

5:22:03just keep living in this little, you

5:22:05know, fish tank that we are the victims

5:22:07of open anthropic on whatever they ship

5:22:09and we just have to be like LM rappers.

5:22:11So I'm going to pitch software research

5:22:13as a direction here. The theme is going

5:22:15to be real software engineering, not

5:22:18just vibe coding. There's it's going to

5:22:20be like the kind of stuff that everyone

5:22:21has known if you've been a software

5:22:23engineer. Decomposition, breaking tasks

5:22:25down. We need to think about stages,

5:22:27ideally ones that can be verified so we

5:22:29have real encapsulation. And we need to

5:22:31actually do state management instead of

5:22:33assuming everything is global state

5:22:35within a KV cache. And the theme is just

5:22:37going to be like real

5:22:38engineering by people who care a lot

5:22:40about the end result. It doesn't have to

5:22:42be valuable. Ideally, it's sick as

5:22:44Um, and you know, we should be pushing

5:22:46the limits. I'm going to go into a

5:22:49couple of like my favorite ideas and I

5:22:51want to figure out how to encourage

5:22:53these in the world. So, this is my first

5:22:55step at this. I'm just going to talk

5:22:56about it. We have raised a bunch of

5:22:58money. So, I would like to figure out

5:22:59how to fund this, but we don't have the

5:23:00operations help to do that. But here are

5:23:02just some ideas out there. Um, I think

5:23:05agents are they're very cool, but like

5:23:09they're they're so inflexible in their

5:23:11design because of the tyranny of the KV

5:23:13cache. I don't have the time for this if

5:23:15this timer is to be believed, but

5:23:17believe me that there's so much that

5:23:18people have tried to do but failed at

5:23:20because it ends up becoming inefficient

5:23:22because it fights against the KV cache.

5:23:24But I think there's a lot more to do.

5:23:26Why is memory o of nan, right? Like why

5:23:28can't we have all of our history of

5:23:30coding agents always included in every

5:23:32new call? That is how continual learning

5:23:35works, right? And we have the memory

5:23:37available. It's just that we don't have

5:23:39the smarts to map produce over the

5:23:40memory to find out like what you should

5:23:43what you should remember, what should

5:23:44you shouldn't. Um I think there's many

5:23:46ways to do login or even better lookups.

5:23:48Why when you're making many agents at

5:23:50once, why can't they not share state?

5:23:52Why do they need to rediscover the same

5:23:54patterns of the codebase over and over

5:23:56and over again? And why can't non-naive

5:23:58routing and sub agents work? Like even

5:23:59naive routing doesn't work, but like why

5:24:01can't we do these things? And why does

5:24:03like the you know the big labs keep

5:24:04going back and forth on whether or not

5:24:05sub agents work? Um I think that like

5:24:08from an engineering perspective there's

5:24:10elegance to it but they fight against

5:24:12the KV cache and I think that if we

5:24:14design agents around that we could do so

5:24:16much more. This one's the the probably

5:24:19the most vague of my directions but I

5:24:20feel very strongly about it. Um the

5:24:23world is full of principal agent

5:24:24problems. This is when you have like you

5:24:26know ulterior motives between this is

5:24:28not a coding agent but like any agent.

5:24:30The the classic example is citizens

5:24:32versus elected officials. I don't want

5:24:34to make a too political claim here, but

5:24:36like there's misalignment in all levels

5:24:38of society. And this results in like

5:24:40waste, fraud, bad things, and it it

5:24:44results in for a society that wants

5:24:45fairness, you need to overemphasize the

5:24:48letter over the spirit, which then

5:24:49results in loopholes and play uh pay to-

5:24:52win dynamics in the world. And I think

5:24:54the root of that problem is we can't

5:24:55encode the spirit. Just like with

5:24:57software, we're constrained to encoding

5:24:59bits instead of what we actually meant

5:25:01in that situation. And I think so many

5:25:04organization, society could be

5:25:05transformed by really encoding the

5:25:07spirit all over the place. Obviously,

5:25:09real-time intelligence security, can we

5:25:12prevent the OpenAI hugging face sandbox,

5:25:14blah blah blah. Um, why did why couldn't

5:25:17they prevent it? It is not because they

5:25:19didn't have the intelligence. I would

5:25:20argue it's a lot easier to figure out

5:25:22they're doing a naughty thing than to

5:25:23actually do the naughty thing. It is

5:25:25because it was not fast, cheap, and

5:25:27reliable enough to actually run real

5:25:29time. And what happens when every

5:25:32corporation has their own real-time

5:25:34security that's actually smart? How cool

5:25:36would that be? I can't talk

5:25:38about generative UIs because people are

5:25:40going to be talking about it afterwards.

5:25:41I don't want to steal their thunder, but

5:25:43it's cool as Trust me. Um,

5:25:46like semantic dependent typing. Why

5:25:48can't we have like a type system that

5:25:52actually is semantic, right? like why

5:25:54can't we like declare what local

5:25:57properties we want so we don't have to

5:25:59read a chain of dependencies deep down

5:26:01these all should be possible and imagine

5:26:03if you could like run a jeike model in

5:26:05your lint checker to check if you're

5:26:07it's all type safe and it could you know

5:26:09uphold semantic properties it would be

5:26:11so cool there's there was a

5:26:15great Twitter video on um like a voice

5:26:17interface in general I think new

5:26:19interfaces are so exciting this is to me

5:26:22the return of like this is going to show

5:26:23by agent software like do what I mean.

5:26:25This used to be like a popular thing but

5:26:27it never really worked because you it

5:26:29was it was too many hard-coded rules to

5:26:31really figure out what I mean. Um

5:26:33there's stopping slop obviously a big

5:26:36theme of here but like we definitely

5:26:38believe this. This is my props. I have

5:26:39to we have a stop geni baseball bat for

5:26:42type safe. It is very very awesome. I

5:26:45want to talk about the root of slop. Um

5:26:47the the root of slop is that AI is

5:26:49unreliable. Therefore, you as a company

5:26:52or you as you know like someone who's

5:26:54doing stuff, it can't be used for

5:26:55decisions that costs you stuff. But it's

5:26:58actually totally fine to push the costs

5:26:59into other people, which is horrible.

5:27:01You know, externalized costs and then

5:27:02this is what ends up becoming stop. You

5:27:05know, if people actually read the stuff

5:27:06and filter it and like care about the

5:27:08quality, it becomes not slop, but now

5:27:10the world is full of this because

5:27:12there's an asymmetric advantage for

5:27:13people who are like bad actors and don't

5:27:15give a So, I think that we can

5:27:17save the world from at least inbound

5:27:19hell among many other things and that

5:27:20would be very cool. Um, and even

5:27:23something I think is super sick is

5:27:25cursed software. Like why if you guys

5:27:28have ever seen the subreddit bad UI

5:27:30battles, it is so cool. Even

5:27:32though they make the horrible interfaces

5:27:34like this whole game of like would you

5:27:35like to delete your account becomes you

5:27:37know a random thing. And and I actually

5:27:40think doing things because we can is

5:27:42what software engineering is about

5:27:44because who knows maybe 1% of these

5:27:46ideas actually lead to something super

5:27:48awesome. And I think like really

5:27:50cultivating even bad ideas is kind of

5:27:53what being like this you know playful

5:27:54hacker is all about and generally just

5:27:58automating everything. This is what the

5:28:01followup this is this is a slop image

5:28:02but uh sorry logo is a little bit wrong

5:28:04but don't stare at that. That's a little

5:28:06weird. Um, this is what a lot of the

5:28:08follow-up talks are going to be about.

5:28:10Thank you. Um, and and if you're a

5:28:13software engineer, you almost certainly

5:28:15think in the world of bits. If you know

5:28:17anything about software, uh, and you've

5:28:19interacted with software, you know that

5:28:21there are things software can do and

5:28:23software cannot do because that is just

5:28:25only obvious, right? Like there's some

5:28:28things that you know you need to do

5:28:30because that is not compatible with

5:28:31software. But what if those constraints

5:28:33were no longer true? Maybe not today,

5:28:35but maybe in like six months or a year.

5:28:37What couldn't we automate? And that's

5:28:39kind of where I want to leave this whole

5:28:41thing because I think there is very

5:28:43likely to be a future where it's not a

5:28:45question of whether or not we can

5:28:46automate something. It's going to be a

5:28:48question of like how much work is it,

5:28:49which is how it is right now for the

5:28:51stuff that can be automated. And that

5:28:52would be cool as So, hell yeah.

5:28:55Thank you. Um,

5:28:57[applause]

5:28:59I'm super overtime, but thank you to the

5:29:02early believers. You know, we had no

5:29:04idea so many people would resonate with

5:29:05this. We thought we were the insane

5:29:07ones, but it turns out like we were

5:29:09actually the majority of the world who

5:29:10realized AI could be so much more. Thank

5:29:13you to like the people who are like

5:29:14really realistic about reliability.

5:29:16Everyone knows I hate benchmarks and a

5:29:18lot of other people are really screaming

5:29:19out like benchmarks suck.

5:29:21They're actually they hurt the people

5:29:23who do not have time for anything other

5:29:25than benchmarks most. It's tragic. And

5:29:27for the people who are pushing the

5:29:28limits of software, like yes. And

5:29:31we want to be there with you helping you

5:29:33all do that. The the dreamers and the

5:29:35people who don't want to be limited by

5:29:37everything that has happened so far. We

5:29:40want to really fight for you and we want

5:29:42this to be as reliable as a fourth logic

5:29:45gate and that is a very high bar and we

5:29:48have like a lot of work to do. So thank

5:29:50you. Thank you.