Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
7:05:28Hey, how's it going?
7:05:29I'm Will Brown.
7:05:30I lead applied research at Prime Intellect,
7:05:32and I hope you are all having a great AI engineer conference.
7:05:35This is always one of my favorites.
7:05:36It's always just so cool to see what everyone's working on and building,
7:05:39and I feel like it is a place where people get to talk about what the future looks like,
7:05:43and start seeing the trends in the industry
7:05:48and seeing the progress and what works and what doesn't.
7:05:51And that's kind of the theme I want to think about for this talk today
7:05:55is what comes next from what we're doing right now
7:05:59and what are some of the things that we've been working on
7:06:02and also just been seeing in research that's unfolding
7:06:05that kind of points at what we think the future of AI might look like
7:06:10at the organization level.
7:06:11We have seen a lot of very cool progress in agents and interfaces and bots and things
7:06:21like that over the past several months, years.
7:06:25But there's always more coming.
7:06:27Things are moving so fast.
7:06:28I think it's a very exciting time to just imagine and speculate and plan for what comes
7:06:35next.
7:06:36I don't know if any of you remember back in the early chat GPT days, open AI had kind
7:06:42of spelled out their framing of what are the different stages of AI progress on the road
7:06:49to AGI.
7:06:51And I think we moved along this faster than we thought we would.
7:06:57We like, I remember 18 months ago I was here in New York at AI engineer giving a talk where
7:07:03where we had gotten reasoners and chatbots had been around for a while and reasoners
7:07:08were the new thing, but we didn't really have agents yet.
7:07:12And now we've climbed pretty quickly through agents and innovators, and the question next
7:07:19seems to be, okay, what happens for organizations?
7:07:23does it mean for an organization to truly be AI native, not just as a startup that uses
7:07:30a lot of agents and builds everything with Claude Code, but at the large enterprise scale,
7:07:36do companies of the future look like that involve deep integration of agents throughout
7:07:42the life cycle of the organisation?
7:07:44And what do we need to build on the path there?
7:07:48So a bit about Prime Intellect, just for some backdrop.
7:07:52We build a lot of different things around AI infrastructure for training, deploying,
7:07:58and continuously improving models, and especially open source models and open source tooling.
7:08:03And so we do a lot of things, and we roughly do the sorts of things that you would do inside
7:08:09of a big lab, and these are our products.
7:08:12And the reason we do this is because we are imagining a future that is coming pretty quickly,
7:08:17it seems, where open source models are readily available to train and deploy on your own
7:08:23workflows, on your own data, in your own harnesses, and can be the sorts of things that organizations
7:08:28organizations can have grow with them over time that allow compounding their institutional
7:08:32knowledge, their institutional advantage from their workflows, from their data, from their
7:08:35people and their processes in addition to using the models from the frontier labs.
7:08:41This talk will be a bit about how we see that evolving and unfolding, where what is the
7:08:46role of training in an AI native enterprise, how does this coexist with the frontier models,
7:08:52and what do we think is going to be the version of this
7:08:56that really unlocks the most value for people
7:08:59and what does it look like?
7:09:01And so organizations have lots of knowledge
7:09:03and the longer organizations have been around
7:09:05and the more established it is,
7:09:07the more implicit knowledge there is
7:09:09that kind of doesn't ever get written down.
7:09:12There are things that do get written down.
7:09:13There's code bases, there's emails, there's processes.
7:09:17Some of these are very explicit.
7:09:18There are rules of you must do this,
7:09:20this thing works like that.
7:09:22But a lot of things live in people's heads and they live in between the lines of emails,
7:09:27of the meeting notes, of the kind of who talks to who.
7:09:30A lot of companies have an on paper org chart and then a real org chart of who actually makes
7:09:36decisions and how does change get adopted throughout the company, what sorts of mandates
7:09:41are likely to take effect.
7:09:43People have built up this knowledge from being at companies for a long time of, oh, this
7:09:47is kind of how things work.
7:09:48This is where the friction points are.
7:09:50And I think in many cases, and I think this is really one of the big opportunities for AI at the organizational level is, is that many individual people are kind of not
7:10:00in a position where they can change the things that are broken in an organization, because no one is actually spelling it out.
7:10:07No one is, like, taking a bird's-eye view of looking at everything that happens in an organization and saying, hey, this over here is inefficient, this could be better, this thing keeps happening across all these different teams, but nobody's figured out that the solution has actually been discovered by that team.
7:10:22And these are the sorts of things that agents are great for.
7:10:25Agents are very capable at combing through large amounts of textual data that's unstructured
7:10:31and finding these kinds of patterns and turning them into something useful and surfacing them
7:10:37as, okay, this workflow has a problem because it is wasting time and it's inefficient and
7:10:43it could be different instead.
7:10:45And we don't want to just do this once, though.
7:10:46We want this to become the way that organizations adopt AI broadly, where agents are able to
7:10:54get the context they need, but also they're able to improve both the agents themselves
7:10:58as well as the organizations.
7:11:00We want agents to be more like our coworkers, but human coworkers learn on the job.
7:11:06Agents currently learn on the job in relatively primitive ways, and they do this via in-context
7:11:12learning.
7:11:13and, oops, sorry.
7:11:20I'm going to go back a few seconds
7:11:21so we can edit this on the recording.
7:11:25Agents learn in relatively primitive ways
7:11:27through, like, in-context learning,
7:11:29and this typically looks like them
7:11:31kind of having a skill file written by somebody
7:11:34or, like, looking around a code base,
7:11:35but they don't get the institutional-level
7:11:38latent knowledge that human employees really do.
7:11:42They relearn a lot, they have to rediscover things, they spend a lot of input tokens.
7:11:46I think nowadays people realize that cached input is the bottleneck in many agent costs.
7:11:53And even if you do get great cache rates and cache costs go down, you still have to reprocess
7:11:57a lot of input to redo a workflow every single time.
7:12:00And so to get around this, you kind of want to bake in more into the model, especially
7:12:06for things that are harder to write down as rules.
7:12:08I like thinking about the spectrum from rules to patterns, as well as conceptually code
7:12:14versus data.
7:12:15And so lots of things live as code, and skills are kind of like this, rules are kind of like
7:12:20this, but things like preferences and friction, heuristics and trends, these are all data.
7:12:25And institutions have so much data that they are running on, and the employees of the organization
7:12:32implicitly learn that models never really get a chance to learn.
7:12:35And this is the sort of thing that we see as blocking them from becoming true coworkers.
7:12:41And so there is a question of what goes in context, what goes in the weights.
7:12:45And part of it is you kind of need to earn it.
7:12:47Things don't just go in the weights right away.
7:12:49But if you see every new generation of models, the models get better not because the prompt
7:12:56is better, but because they train them more.
7:12:58Training is useful.
7:12:59Training works.
7:13:00Training is why we see the progress we do.
7:13:02and the model is roughly trained on lots of data
7:13:06that is aiming to approximate things
7:13:08that the labs expect will be high-value,
7:13:12repeatable workflows at organizations.
7:13:14And I like this.
7:13:16I think there's a process that currently it works.
7:13:18It's how the models get better.
7:13:19But if you spell it out, it's relatively inefficient.
7:13:22So the way that a model would learn how to get really good
7:13:25at using some combination of Salesforce tools, for example,
7:13:29Let's say someone who knows how those tools get used has to be looking for work.
7:13:36And maybe they previously worked at a place that were using these tools and now they are
7:13:40browsing expert network sites to make money in their free time.
7:13:44And they are sharing their knowledge to someone who works at an expert network company.
7:13:48This person at this network company will then take this data from the human and pass it
7:13:54to someone else whose job is to create RL environments.
7:13:56This RL environment might get sold then by someone who doesn't have this expert knowledge
7:14:02to a researcher at a lab who also really isn't the expert in the domain, who then will be
7:14:06refining it to put it into a training pipeline to go into the next frontier model run to
7:14:13make a model that has kind of through this game of telephone gotten some knowledge of
7:14:18that human originally.
7:14:20But there's so many steps in this pipeline and it's a relatively secret pipeline.
7:14:23People talk across two stages of it, but the human who knows this thing is not supervising
7:14:29the training run, is not directly inputting how the model gets trained in the way that
7:14:34a human could if they were doing it for their coding agent that they're working with day
7:14:37to day.
7:14:38And so what we see as an opportunity, and also where we would hope to go, is to close
7:14:44this gap and to allow the person who, the same way that when we work with our coding
7:14:48agents, we're prompting them the same things every time, telling them to not repeat the
7:14:52same mistakes.
7:14:54This is how you could train models.
7:14:56You can tell the models things and they will learn these things durably, not just as a
7:15:02line in your agent's MD which accumulates and you have to refactor it all the time.
7:15:06You can do some of that, but a lot of times what you really want is you're giving contextual
7:15:10preference.
7:15:11In this setting, this thing that you did was not exactly what I wanted and I would have
7:15:15preferred this instead.
7:15:16And we want systems that can compound this into durable learning.
7:15:20And training is a very effective way of doing this, because the shape of data you're giving
7:15:24there is quite close to what you would want from an actual training pipeline.
7:15:33And so there's several different types of learning.
7:15:35There's things that people do, there's things that you can do at the harness layer with
7:15:38skills and memory.
7:15:40There's knowledge that just lives throughout an organization in file systems and databases
7:15:44that you retrieve with tools and with code.
7:15:48Then there's the models themselves which have this really broad world knowledge.
7:15:52But we all have, I'm sure, experienced frustration when using a model whose cut off of knowledge
7:15:59is a while back.
7:16:01You can't really talk to the models about the news of the past six months nowadays.
7:16:05They have to go relearn a world model in context every single time.
7:16:10This does not allow them to really operate as fluently as they can discuss things that
7:16:15happened longer ago.
7:16:16So, organizations, to get the most value out of models, we imagine they will want to increasingly
7:16:22have models that really understand what is going on in the organization at an intuitive
7:16:27level.
7:16:28This is the sort of thing that you want to train for.
7:16:31So how do you get there?
7:16:32What is the road for an organization to go from having this idea in the abstract of we're
7:16:39going to train models on our data to actually getting there?
7:16:42And I think one really useful starting point is evals as essentially test cases.
7:16:46I think, like, with code, we've gotten to a point of maturity around, like, testing and life cycles
7:16:52and software development life cycles for ensuring that code bases grow healthily over time, usually.
7:16:59And agents are doing so much of this work for us.
7:17:02And the version of continual learning that I think is, like, most practical for most organizations
7:17:07is just treating a model kind of like a code base,
7:17:10where it is the sort of thing that you can have a life cycle around.
7:17:14You can add test cases to.
7:17:15You can add features to.
7:17:16You can add assets to.
7:17:19And this is the sort of thing that the process of using agents,
7:17:23like initially you collect traces.
7:17:24These traces have your prompts
7:17:26and they have your feedback to things that models did.
7:17:28And this, just from the process of using these models,
7:17:32you are accumulating the assets you need to create evals.
7:17:36It's just that making evals historically has been a little bit painful
7:17:39and a lot of organizations have not necessarily wanted to do it manually,
7:17:43but the agents are quite quickly getting to the point where,
7:17:47thankfully, we have robots that can do a lot of the eval curation for us,
7:17:51especially as we train them more on the best practices of making evals
7:17:55in ways that are robust and amenable to learning.
7:17:59And so I think another useful example here is the decision model craze
7:18:03that has kind of been taking the world by storm in the past couple weeks with Jev,
7:18:07and also the parallels around routing
7:18:09and people being very excited about cost optimization
7:18:11because we see that it's possible to spend
7:18:14very large amounts of money on Claude every month
7:18:16and we figure out when should we use a cheaper model.
7:18:20And this is the sort of thing that I think people are recognizing
7:18:23needs to kind of live in the harness.
7:18:25But it also is an attribute that looks relatively similar
7:18:28to decision models.
7:18:30And the reason I bring both these up together
7:18:31is because there's things with very clear labels
7:18:33where there's a right answer
7:18:34and there's an opportunity for a human or another agent to give feedback.
7:18:38You can have the big model and the small model do the same task,
7:18:41judge whether the small model was as good as the big model.
7:18:44If so, then this is now data of, like, for this kind of thing,
7:18:47we can start routing to the small model more.
7:18:49Same for a decision model that's learning to kind of mimic the way
7:18:52that a human in a certain situation would make a judgment.
7:18:55This is a great labelable action,
7:18:58where the human might be doing the approval
7:19:00the same way that, like, we approve our tool calls and Claude Code sometimes,
7:19:03times, but I think most people have turned on auto tool acceptance, but every organization
7:19:09for more bespoke things that aren't just about tool security are going to have their own
7:19:13rules and preferences, and these are the sorts of things that you could then compound into
7:19:17a decision model.
7:19:18And so this is a very believable form of continual learning that a lot of people could just plug
7:19:22in tomorrow and see immediate value from.
7:19:26But more broadly, what we see people wanting to do, and this is a lot of the work that
7:19:30we do at Prime Intellect is working with organizations that are looking to do relatively
7:19:34ambitious training projects around turning their traces from deploying a model in production
7:19:40for a high-volume use case into rich, robust environments that are blended, real, and synthetic
7:19:46that kind of capture the complexity of the real tasks they want to be using long-running
7:19:51agents for.
7:19:53And agents are very useful with sufficient orchestration around them, sufficient tooling
7:19:57and humans in the loop at turning these into
7:20:00environments that are rich enough to train in.
7:20:02And we see that this is getting more and more automated with every new generation of open and closed models.
7:20:08And this means that we have an opportunity to start having this whole thing of taking the stuff that we're already doing and turning it into a self-improvement loop just kind of fall out magically from applying agents to curation of the data
7:20:22and creating interfaces for humans to give signal about what they want.
7:20:27This is an idea that I've been, for my own setup,
7:20:30been playing around with a lot
7:20:32that I think will become a popular interface
7:20:35for interacting with models more in the future.
7:20:37You're seeing it a little bit in some of the bots,
7:20:39but I think, especially now that models are good at writing code on the fly,
7:20:43these kind of widget card features
7:20:45that are, I think, a very practical form of dynamic UI
7:20:49that are also really useful for eliciting human decisions.
7:20:52So they're kind of like the tool call approval thing, but much more personalized, much more
7:20:56specific, and also kind of get around the problem of, like, okay, you have 20 different
7:21:02bots that are going, and you have to remember which ones are where, and who is curating
7:21:06all this for you, and then people are like, do you want one bot?
7:21:09And then you're like, oh, one's great, but I actually want two.
7:21:11Then you have two.
7:21:12Then you have 20.
7:21:13And I think we kind of are oscillating back and forth between this single stream and the
7:21:17multi-stream agent threads.
7:21:19But what we really want to do are make decisions.
7:21:20We want to give ideas to the agents, let them run with the ideas, and then come back to
7:21:25us when we need to make decisions.
7:21:27And the agents hopefully should make good choices about when to bring us decisions.
7:21:31And if agents can create interfaces on the fly for us that allow us to very directly
7:21:36give them the information they need, one, this is much more ergonomic for us as humans
7:21:43than reading, trying to parse through a long thread message to see where was the question.
7:21:47It's also kind of like the human dual side of the decision model, Jev style, but it also
7:21:54is perfect data.
7:21:55And so it's a setting where we've been experimenting with this as a way of training models to kind
7:22:01of get this feedback loop going from having human input with these kinds of natural interfaces
7:22:06be the sort of thing that directly acts as a label and is the sort of thing that a model
7:22:11can directly improve on because there is a very explicit target.
7:22:14and it gets around a lot of the problems
7:22:17that we kind of hear about in reward hacking and whatnot.
7:22:20Those things are worth addressing too,
7:22:21but this is where we see a very useful starting point
7:22:24being for a lot of these problems.
7:22:26And this can be done,
7:22:28it doesn't have to be with every model in your stack,
7:22:29it can be the orchestration layer models,
7:22:31it can be the decision models,
7:22:32but it is an entry point for
7:22:33kind of starting to have this compounding
7:22:35of preferences and latent knowledge
7:22:37happen in the models.
7:22:43And so I think there's a recent example today
7:22:45I just added this earlier
7:22:46From a company called Paper Instruments
7:22:49They use some of our tooling to do this
7:22:51They're friends of ours
7:22:53But they trained a model for 300k
7:22:56That is kind of Pareto Frontier optimal
7:22:58For knowledge work for things like spreadsheets and slides
7:23:02And it's like an employee scale investment
7:23:04For an org to do
7:23:06where you could go hire a software engineer
7:23:09or you could train a model.
7:23:11And you can also spread this out
7:23:12over a longer period of time
7:23:13if you're kind of compounding the knowledge over time.
7:23:15But it's the sort of thing
7:23:16that is increasingly becoming quite practical
7:23:19and it's the sort of thing
7:23:20that a lot of organizations we see wanting to do now
7:23:22and we kind of see the trend
7:23:24of how much more the demand for this
7:23:26is growing month after month
7:23:29as people start recognizing,
7:23:31one, the value of this
7:23:32and how kind of the open models
7:23:34are good enough that you can get them to be much better and also much more personalized
7:23:39by turning your knowledge for your domain, your harness, your environment into a training
7:23:45setting for them.
7:23:47But also it's the sort of thing that naturally lends itself to a learning loop.
7:23:51And so this is a prototype of something we're doing internally around kind of how we're
7:23:55imagining people can really deploy continual learning in the world, which is that you can
7:23:59have traces, you can start collecting these, and you can then, like, every night, see the
7:24:06proposed experiment, look at the things that have been flagged as issues, and kick off
7:24:11a training run for the night.
7:24:11Maybe your inference capacity in the day is serving your agents.
7:24:14At night, employees are not using them as much, and you can use that capacity for training
7:24:19instead.
7:24:19And so this is a very natural way that we see organizations able to, like, quite soon,
7:24:24actually do for real continual learning in the weights of the models and start compounding
7:24:29their knowledge into better and better capabilities for their agents.
7:24:35And so this is kind of where we see a lot of this culminating towards, there's so many
7:24:39things you do first.
7:24:41You discover your workflow, you create skills, you make evals, you do routing, and then eventually
7:24:47you start to want to get the economies of scale, get the compounding benefits that come
7:24:52from the data inside of your companies, your organizations, and have the organizations become
7:24:57increasingly self-improving in ways that get better and better over time.
7:25:03That's what I got.
7:25:06I can maybe take questions if there's any time.
7:25:08I don't know.
7:25:08No questions.
7:25:10And, yeah, we're at Prime Intellect.
7:25:12We are working on a lot of this stuff.
7:25:14A lot of it is open source and public, and we blog about it and tweet about it, but also
7:25:18we work with companies and we're hiring and growing, and so get in touch if you want to
7:25:21get in touch.
7:25:22Thanks.
7:25:22Thank you.
