AI Engineer World's Fair 2026
Scaling to Long Horizons
Read the talk
Scaling to Long Horizons
From Galactica’s base-model demo to agents that learn across extended tasks, long-horizon reasoning depends on memory, learning signals, realistic environments, and compute working together.
From a talk by Ross Taylor and Chengxi Taylor
Before you start: Basic familiarity with language-model pre-training and reinforcement learning will help; compaction, critics, policy lag, and bootstrapping are explained as they arise.
What turns a capable model into a usable product?
What turns a capable language model into something people can reliably use? Ross Taylor’s route into that problem began with Papers With Code. He describes a London startup acquired by Meta in 2019, followed by a transition into research on Galactica, then post-training for Llama 2 and Llama 3 after ChatGPT appeared. By the time of this talk, he and Chengxi Taylor were leading General Reasoning, pursuing the algorithms, environments, and compute needed for longer agent tasks. That history also shaped Ross’s conviction that small, focused teams can make substantial contributions even when model development demands enormous scale.
The product lesson became painfully concrete in 2022. Galactica appeared roughly two weeks before ChatGPT, whose initial system was based on GPT-3.5. Both had capable underlying models, but Galactica shipped a base-model demo. Its willingness to continue a prompt included hallucinations and compliance with unsuitable requests. ChatGPT added a crucial reinforcement learning from human feedback pipeline that shaped how the model responded to users.
A good base model is not enough. Ross treats the two launches as a personal natural-experiment analogy: similar underlying promise, very different public outcomes. It is not a controlled comparison isolating RLHF, but it explains why his attention shifted from base-model capability toward the training that makes that capability useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Strong results did not solve the behavior problem
Evidence for that distinction was already available in InstructGPT. On the study’s prompt distribution, human evaluators preferred outputs from 1.3B-parameter InstructGPT over those from 175B-parameter GPT-3. Ross rounds the smaller model to one billion parameters in the talk. The result concerns human preference after instruction tuning and human-feedback training, rather than superiority across all tasks or an isolated measurement of RL alone. A much smaller model could better satisfy the people using it.
Galactica’s public demo exposed the opposite problem. Users could request a research paper about Dyson spheres or a report claiming benefits from eating crushed glass, and the model would produce text in the requested form. Ross recalls the backlash—and the Meta association—as overshadowing the technical work. The failure was consequential precisely because a scientific-looking answer could be generated without the behavioral judgment a public product needed.
Meanwhile, Ross reports strong scientific-domain results against PaLM, Chinchilla, and GPT-3.5 with less compute. The equation result has a more precise scope: on manually evaluated zero-shot LaTeX-equation generation, Galactica 120B scored 68.2%, versus 49.0% for text-davinci-002. That exact comparator matters; this was not an evaluation against the ChatGPT product. Ross also recalls a chain-of-thought comparison of roughly 36% for Galactica 30B versus 19% for PaLM 540B, without naming the benchmark in the spoken passage. These domain comparisons explain why the demo’s reception could not be reduced to a weak underlying model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Curated data and computation before the answer
Galactica also supplied lessons about how to obtain those capabilities. Ross contrasts its curated 105B-token corpus with roughly one trillion tokens for Chinchilla. His emphasis is on the quality of the source data, not simply accumulating more tokens. Repeated training over that data challenged the contemporary convention of stopping after one epoch. He connects this experience to the later rule of thumb about four epochs of repeated data; that is a historical connection, not a claim that four epochs is always optimal. Corpus size and total exposure during repeated training are different quantities.
The next contribution concerned what happens between the question and the answer. Several approaches were emerging at the time:
| Approach | Role of intermediate generation |
|---|---|
| Chain-of-thought prompting | Ask for the steps leading to an answer |
| Numerical scratchpads | Record intermediate calculations |
| Galactica’s thinking tokens | Delimit an internal working-memory process |
Ross presents Galactica’s tagged working space as an early way to make the model spend inference compute before answering. The practical distinction is that intermediate tokens become a place to work through the problem, rather than merely a stylistic request to explain the final response. He regards the data, repeated-training, and thinking-token ideas as pioneering contributions; their technical usefulness does not depend on resolving historical priority.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimizing the work between thinking tags
After Galactica, the team took on Llama post-training. Ross kept pursuing a narrower question: what if reinforcement learning optimized the intermediate work inside the thinking tags? He connects that intuition to later DeepSeek reasoning models and OpenAI o1, but the starting conditions were very different. Llama 2 had weak mathematical training data and a context window of roughly 4,000 tokens, leaving limited space for extended reasoning.
The unpublished training recipe addressed both the starting model and the learning objective:
- Continue pre-training on mathematics and science. Improve the base model’s ability to represent and solve the relevant problems.
- Apply PPO with verifiable rewards. Optimize against outcomes that can be checked.
- Initialize the value model from a strong outcome reward model. Give the critic a useful starting point rather than learning it entirely from scratch.
Ross explicitly distinguishes this recipe from GRPO: the value model was a central part of their PPO setup. He reports internal state-of-the-art math and reasoning results, but those experiments were unpublished.
Better scores did not produce everything the team hoped to see. They did not observe inference-time scaling or the characteristic reflection and backtracking later associated with DeepSeek-R1 and o1. An objective that improved answers had not yet produced a model that reliably used additional thinking to reconsider its approach.
When later reasoning models appeared, Ross revisited that gap. His explanation combines better base models, more RL compute, and larger context windows. An apparently similar research idea can behave differently once those prerequisites change. This is also a point about access: having a GPT-4-level model earlier can expose behaviors and research directions that remain invisible with weaker models. He presents this as a retrospective scaling lesson, not a controlled demonstration of which ingredient caused reflection.
That experience did not convince him reasoning was solved. It motivated the formation of General Reasoning to pursue larger tasks, and it sets up Chengxi’s next question: what changes when useful reasoning must continue far beyond one response or one context window?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Continuing beyond the context window
Cancer research, Millennium Prize problems, and reaching Mars are Chengxi’s motivating ambitions, not capabilities demonstrated here. Their common requirement is sustained work. A system meant to contribute to such problems must preserve progress through reading, experimentation, failed attempts, and new ideas rather than treating each response as a self-contained task.
She uses Fermat’s Last Theorem as an analogy: years of reading papers, writing scratch work, and stepping away to develop ideas. Her illustrative conversion puts such intellectual activity at tens or hundreds of billions of tokens, against the million-token context windows she uses as a contemporary reference point. This is an intuition for the scale mismatch, not a measured token count or a universal ceiling on context length.
Compaction lets work continue despite that mismatch:
- Generate within the available context.
- Summarize the accumulated context as the window fills.
- Carry that summary into fresh context and continue generating.
The summary becomes part of the state on which later decisions depend. Chengxi proposes applying reinforcement learning to both the task policy and the compaction behavior, so the system learns not only how to act but also what to preserve for subsequent work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning from long trajectories and retaining their history
Continuing generation does not by itself make a long trajectory easy to train on. Chengxi identifies three optimization problems:
- Gradient variance: longer trajectories increase the variability of the learning signal.
- Sparse rewards and credit assignment: a result arriving much later gives limited direct information about which earlier decisions helped.
- Variable trajectory lengths: episodes of different lengths complicate optimization.
These problems concern learning from an extended sequence of actions, even if memory is sufficient to keep acting.
A critic, or value model, estimates how promising a state is. Chengxi proposes it as a way to reduce variance, work naturally with compacted trajectories, and encourage batch diversity. It also supports bootstrapping: using an estimate to obtain a learning signal before the episode ends. The cost is an additional model to train alongside the policy, making the setup more complicated than GRPO. The critic’s role in keeping training moving becomes especially important when episodes are too long to wait for their final rewards.
External memory provides complementary ways to retain information:
- Filesystem tools act as a scratchpad for reasoning that need not remain in the current context.
- Self-search tools retrieve material from the agent’s previous trajectory.
- Archive tools preserve results that later work can build on.
Archives require a task-specific boundary. Reusing an earlier result can be the point of automated research, where knowledge should accumulate. In another evaluation, retrieving an existing answer could bypass the reasoning the environment is supposed to test. Memory access therefore changes the definition of the task, not just the amount of information available to the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A season of decisions, not a single correct answer
How well do frontier models handle this kind of extended task? General Reasoning built KellyBench, which Chengxi says attracted Financial Times front-page coverage. Agents build machine-learning models for trading on Premier League football matches over a one-year horizon. The appeal is an objective with financial meaning: sports betting connects forecasts and decisions to returns rather than stopping at a competition score. The suggested commercial upside is motivation, not a demonstrated outcome.
In the presented KellyBench evaluation, all evaluated frontier models had negative average returns from a normalized £100,000 starting bankroll in a historical-market simulation. The distinction matters: this was not live money lost by deployed agents, and an average loss does not mean every run lost money. The result remains a demanding test of whether model-building and repeated decisions add up to a successful season; the current public leaderboard has subsequent updates and should not be substituted for the talk’s results.
Chengxi’s diagnosis begins with the industry’s emphasis on coding and procedural work. A narrowly specified repair or implementation task often leaves relatively few acceptable solutions. Success there need not establish the ability to search creatively through a much larger space of possible strategies.
Open-ended environments add two further requirements. First, the agent must handle uncertainty and complexity that a tightly specified task may remove. Second, it must operate among other actors with different beliefs and objectives. Chengxi argues that current benchmarks and simulations do not capture enough of these interactions. A longer sequence of procedural instructions alone would not supply the real-world fidelity she wants.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping GPUs busy while episodes continue
Long horizons also change how scarce GPUs should be divided between inference and training. In a sequential RL cycle, inference produces trajectories, then training updates the policy. As trajectories lengthen, waiting for generation to finish can leave training resources idle. Pipelined RL overlaps those activities: training starts on available sequences while additional sequences are still being generated.
The overlap creates off-policy data. A trajectory can have been generated by an older policy than the one currently being trained. Chengxi reports that, in their experience, up to eight steps of policy lag is normally acceptable. That is an empirical tolerance for their setup, not a general bound. If inference were to take weeks, ongoing updates could exceed that tolerance before a trajectory completed, forcing training to wait again.
Bootstrapping offers another route. Before the episode finishes, the value model estimates the expected outcome, allowing training to use a signal without waiting for the terminal result. Chengxi likens this anticipatory signal to dopamine; the operational mechanism is an estimate of future value. Better utilization now depends on the accuracy of a learned prediction, introducing value-model bias.
| Training approach | What enables an update | Main cost |
|---|---|---|
| Wait for completed rollouts | Finished trajectories | Idle time during long generation |
| Pipeline generation and training | Available sequences while others continue | Policy lag |
| Bootstrap with a critic | Estimated value before completion | Value-model bias |
These choices connect systems scheduling to the learning algorithm. Keeping hardware occupied is useful only if the resulting training signal remains good enough to improve the policy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Environments for sustained progress
The environment infrastructure must support the same ambition. Chengxi introduces OpenReward as General Reasoning’s shared platform for RL environments. At the time of the talk, she reported more than 350 environments accessible through a single API endpoint. She says the company uses it for internal RL and that frontier and newer labs also use it. The infrastructure proposition is straightforward: a common access layer makes many environments available to the training effort.
The progression from pre-training to reasoning and agents leaves a connected set of problems. Longer tasks need memory that preserves useful work, algorithms that can learn before distant rewards arrive, environments that expose uncertainty and competing actors, and compute schedules that do not spend the horizon waiting. None of the proposed techniques removes every trade-off: compaction changes what is retained, critics add learned estimates, and pipelining permits stale-policy data.
Chengxi closes by returning to patience. If the goal is to help solve problems that already demand years of human effort, sustained work must be part of both the agent’s design and the research program building it. That ambition is the mission she attaches to General Reasoning: advancing algorithms, environments, and compute together so that progress can continue across much longer horizons.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The InstructGPT study explaining supervised instruction tuning, human feedback, and preference evaluation.
An overview of season-long simulated football betting, agent strategy evaluation, and benchmark results.
Environment access and evaluation instructions for running agents through a simulated Premier League season.
A catalog of reinforcement-learning environments with shared API access and rollout examples.
The original report distinguishing R1-Zero's reinforcement learning experiments from R1's multistage training.
Read the complete timestamped transcript
- 0:00
[upbeat music] Uh, so this talk is called Scaling to Long Horizons.
- 0:15
My name's Ross. Uh, I'm the CEO of GR. We're a London-based reinforcement learning company. Uh, before GR, I was the reasoning lead [REDACTED:username] Meta AI working on Llamas, uh, Galactica, lots of other models back in the day.
- 0:27
I'm joined by Chengxi, uh, co-founder and president of GR. And yeah, h- today we're gonna talk about algorithms, environments, compute, all the things you need to do to get agents scaling, uh, to kinda longer tasks.
- 0:41
So we're gonna have two parts to this talk today. I'm gonna first of all start with a personal perspective about, you know, the early days, the golden age of language modeling between, like, maybe twenty-twenty and twenty-twenty th- three.
- 0:52
Uh, I'll talk about, like I said, all those models and some of our early reinforcement learning efforts for LLMs. And then Chengxi is gonna talk about, you know, what's ahead.
- 0:58
You know, what are the next frontiers, and yeah, that's gonna be a really interesting talk with a lot of alpha, so I'd encourage you to stick around for that.
- 1:06
So the journey so far. So my journey started here. Um, so this was the Papers With Code team. I'm sure many of you used Papers With Code back in the day.
- 1:15
So we were a London-based startup, twenty nineteen. Uh, we were acquired by Meta later that year, and then we had a crazy transition within Meta to do research. Um, so we did, like I said, Galactica.
- 1:26
Then after ChatGPT came out, we started the post-training for Llama 2, Llama 3. So all the great work you, uh, saw there was folks in this room, and lots of other interesting stuff that never got published as well.
- 1:37
Uh, Reasoning Llama, and lots of other things. Um, so yeah, this, this small team, I like to think, like, the open weight kind of revolution started in this room.
- 1:44
And, you know, it really, like, hit home this idea to me that kind of small-focused teams, like, even in, like, the age of scaling, can do amazing things if people are aligned.
- 1:55
Now, for me, uh, things got particularly crazy in two thousand and twenty-two. Uh, so let me tell you a story. Um, the media perception is that ChatGPT came out of nowhere, you know, shocked the world, and that's how kind of the mo- modern AI wave started.
- 2:10
But, you know, I have a different personal perspective on this because two weeks before ChatGPT came along, there was another language model called Galactica. So let's talk about Galactica.
- 2:20
Galactica and, you know, ChatGPT, you know, they were both, you know, in some respects quite similar. They were both based on pretty good base models. Galactica itself was a base model, and then ChatGPT was based on GPT-3.5.
- 2:33
But there was a clear difference in outcomes. So Galactica [REDACTED:username] the time shipped with this, like, base model demo. And as you guys know now, like, base models, they come with a lot of quirks.
- 2:42
You know, they hallucinate. You prompt them to do, you know, silly things, they will do silly things. Whereas ChatGPT wasn't just a base model but had this, like, crucial reinforcement learning from human feedback pipeline.
- 2:54
And this was the key thing that made LLMs, like, really products for the first time. So I'd like to think in a weird kind of way, this is like the first, like, kind of natural experiment showing you that kind of RL, like, provides value, right?
- 3:05
Uh, and to my misfortune, it was, like, a very personal, like, kind of a natural experiment. You know, Galactica blew up. Uh, but that's, like, a good, like, kind of lesson there.
- 3:13
A good base model is not enough. So I took that lesson quite early on. So like I said, RLHF made LLMs products. They were the thing that kinda made LLMs cross the Rubicon into something that wasn't just a toy, but used by now billions of people.
- 3:27
But you didn't have to wait until ChatGPT to see this. Like, even [REDACTED:username] the time, like InstructGPT in twenty twenty-two had these pretty stunning results. Like a one billion parameter model with RLHF was outperforming a hundred and seventy-five billion model.
- 3:40
So two orders of magnitude fewer parameters, but getting better results. So that was astonishing. So if you were paying attention closely, you know, maybe we should've been as well, but we were focused on a, you know, bloody base model, which was hard work in twenty twenty-two.
- 3:52
But that shows you how, you know, important even, you know, basic RL is.
- 3:57
And the Galactica demo itself, I mean, it set off a storm. So ancient history now, but we put out a demo. We let people play around with it. Um, we thought it was kinda cool.
- 4:05
And [REDACTED:username] the time, people got scared. So it was like, you know, like I said, you prompt it on, like, a research paper on Dyson spheres or a report on the benefits of eating crushed glass, and people are like, "Oh, my God."
- 4:16
Um, so that was the state of things in twenty twenty-two. And yeah, I'll be honest, the Meta association didn't h- you know, help us either. Um, and yeah, the tragic story in a way was a lot of the, you know, novel work was maybe overshadowed.
- 4:28
But, you know, the paradox of this whole thing is that [REDACTED:username] the time, Galactica was actually a bloody good model. Um, like it outperformed PaLM, Chinchilla, GPT-3.5 with a lot less compute in scientific domains.
- 4:38
It was state-of-the-art. So again, that also shows you how powerful RL is. You can have a SOTA base model, but that is not enough. Um, so here you see on, like, maths, it was kinda beating Chinchilla.
- 4:49
Uh, kinda LaTeX equations, you know, science, you know, was getting around sixty-eight percent compared to GPT-3.5, forty-nine percent, so crushing there. And chain-of-thought as well. So, you know, a PaLM [REDACTED:username] the time, which was a Google Brain model, five hundred and forty billion.
- 5:01
You know, its thirty billion in Galactica was getting, you know, thirty-six versus nineteen percent, so double the performance, order of magnitude less results. So that, again, reinforces base-- good base models, not enough.
- 5:12
But it introduced some key ideas which I think are very important. I mean, Galactica was the first LLM to really crack data efficiency. A hundred and five billion pr-- uh, token corpus compared to, you know, a trillion, uh, tokens in Chinchilla.
- 5:23
And it was, uh, really contrarian [REDACTED:username] the time because, you know, [REDACTED:username] the time everyone was like, "Okay, we just need more tokens." And Galactica said, "No. High quality, you know, curated datasets really matter."
- 5:32
And that was a real driver of those results you just saw. And it was also, like, the first major LLM to really crack multi-epoch training. It sounds ridiculous now, but [REDACTED:username] the time the consensus was you don't do more than an epoch.
- 5:42
Um, but this kind of rule of thumb, you may have heard of it, like four epochs of repeated data, that was formalized later, but Galactica was the first, like, real empirical result for that.
- 5:51
Now perhaps more importantly, there's this idea of thinking tokens, and some of you might remember this, but it was like really quite buried within the paper. So around this time there were like different ideas for reasoning.
- 5:59
There was chain-of-thought, which was one idea. It's where you prompt for like kind of like the steps. There was scratch pads where you just like put like very like numerical kind of intermediate steps.
- 6:08
But Galactica was really this first idea which said, "No, this is an internal working memory process. This is an internal thinking should be inside these tags, and you should spend the inference compute before you get to an answer," right?
- 6:19
So these are all like quite like prescient ideas.
- 6:23
But, you know, Galactica came and went, blew up. And then, you know, we were kind of tasked as a team to kind of spin up the post-training effort for Llama.
- 6:30
But I had like a personal obsession, which was like reasoning. And I had like a really simple idea [REDACTED:username] the time, which is what if we applied kind of reinforcement learning pressure to this like kind of thinking tags, this work?
- 6:42
Like, what if we just optimize the thing in between the thinking. And if that sounds familiar, then this is like kind of what DeepSeek [REDACTED:username] oh-one ended up doing two years later.
- 6:51
But there's a key difference. So [REDACTED:username] the time, we only had Llama 2 base models, terrible mathematics corpus, terrible results on math. And, you know, the context window. You know, we're all like, you know, very, you know, context r-rich now, one million tokens.
- 7:03
You know, back in the day it was, uh, four, four thousand, which wasn't too fun.
- 7:07
But we still had a recipe [REDACTED:username] the time, and this was unpublished, but it was really good for the Meta. So our recipe was this. Number one, continue pre-training on Llama 2 towards mathematics and science data.
- 7:18
So that's the first thing. Llama 2, shit math corpus, so let's fix that. Number two, PPO with verifiable rewards. But notice this isn't GRPO, right? So we [REDACTED:username] the time had a strong outcome reward model to initialize the value model.
- 7:31
That was a key thing. Lots of data on value models [REDACTED:username] the time. And internally [REDACTED:username] the time, this kind of recipe led to state-of-the-art results on math and reasoning.
- 7:39
So we were kind of like, "Wow, this is like really shows the power of having the right objective." But the really fascinating thing is like we had great results, but we didn't have like inference time scaling.
- 7:51
We didn't have this reflective behavior that became like the hallmark of R1 and oh-1. You know, butt weights, you know, backtracking, all this kind of stuff. So it begs like the question, like why?
- 8:00
Why didn't we have that moment? And we got an answer around like two years later. So there's a couple of like things going on here, but essentially better base models were the thing that really got RL cooking.
- 8:11
And when DeepSeek came out, I was kind of shocked [REDACTED:username] the time. I was like, "Holy shit, we just like tried the same thing. We didn't have this. What's going on here?"
- 8:17
And in a weird kind of way, the real lesson was it was just like the bitter lesson, like the most purest form of bitter lesson possible. Like better base models, more RL computes, bigger context windows, and that's all you need for this kind of emergent behavior.
- 8:31
It also like says something like quite important about the sociology of like research because the fact that OpenAI had this model, you know, GPT 4 level model before anyone else, it allowed them to see further, right?
- 8:41
So that's a really interesting point. Like the, the age of scaling means that if you have certain prerequisites in place, you become smarter. You see further. You see more ideas.
- 8:49
So a really interesting point. So this was me in 2024. I was a very sad panda, uh, defeated by ChatGPT in oh-one. But I wasn't deterred, um, so I wanted to seek the next wave, and I still was like convinced that kind of reasoning hadn't been solved.
- 9:05
So we started GR, uh, to take on truly like big tasks. And with that in mind, I'm gonna hand over to Chengxi, who's gonna talk about what we're kind of thinking about now.
- 9:13
Ross, Ross, Ross. Give me the... [laughs] Hi, everyone. I'm Chengxi Taylor, co-founder and president, General Reasoning. I'm gonna share what it takes to scale to long horizon.
- 9:32
First, I wanna make it clear, long-horizon task is not just an engineering problem. It is a mindset. If we want to solve humanity's biggest problems, such as cure cancer, solve millennium's prize problem, or going to Mars, we have to be patient.
- 9:51
It take time. And if we want AI to move us towards that level impact, we have to think about long horizon.
- 10:02
But here's the first problem. We have scarce context window. If you take Fermat's Last Theorem as example, what it take for the mathematician was over ten years' time of reading paper, writing thoughts in the scratch pad, or taking a walk to generate creative ideas.
- 10:21
If we convert to token, that's probably tens of billions or even hundreds billions. But where we are now, just a one million token context window.
- 10:32
So one solution is use compaction. So what it essentially does is generate a token until the end of the context window, summarize, and then on top of that, generate more tokens.
- 10:45
And the beauty of applying RL in this situation is kind of like kill two birds with one stone. You apply RL to the compaction and also the task.
- 10:56
But here's the problem. With long horizon, there are three issues. The first is the gradient variance scales with the length. And the second is a sparse reward. And you have this, uh, credit assignment problem.
- 11:09
And finally, there's also variable length of the trajectory that adds to the problem of optimization.
- 11:17
So to solve this issue, we can apply critics, which is the value model. And value model can reduce the variance and also have a couple advantages, such as on the trajectory level, it fits compaction very well and also encourage the batch diversity.
- 11:34
And also, I'll talk later on bootstrapping. Basically get signal before the end of the episode. But the downside for this is that un- it's more complicated than GRPO, and basically you have to train another value model alongside with the policy model.
- 11:52
And there are some tools to help with the context limitations, such as a file system tools, which essentially like a scratch pad for AI to write the, uh, reasoning thought, and self search tools which allows agent to search over the previous trajectory.
- 12:08
And then you have archive tools. In the case like auto research, you can build upon your previous result, but we have to be careful. In other scenario, you don't want AI to cheat by just to grab the previous answer without thinking.
- 12:24
So how good are the current model on this long-horizon task?
- 12:31
In the General Reasoning, we construct the, uh, benchmark called a Kelly Bench, where we're actually featuring the front page of the Financial Times. It kind of caught us off guard how much the mainstream have interest in this.
- 12:43
So basically what we did is that we allowed the agents to build machine learning models to, um, trade in the football matches over a one-year horizon. In this case, it's a Premier League, if you're interested in football.
- 12:58
And we're so fascinated by this because there's real money to be made, and if it was successful, it could make billions. There's a whole industry on sports betting, and unlike things like a Kaggle competition, this has a real world implication.
- 13:15
But here's the result. As you can see, we gave all the frontier models 100K to start. All of them lost money.
- 13:27
Sad. And that captured the public's imagination, "Oh, AI is not as great as they thought." [laughs] And why are models so bad [REDACTED:username] long horizon? First, I believe now the AI industry is a little bit too biased towards coding and procedural task.
- 13:45
What I mean is that the current task is, um, most formulated like, do this and fix that. Normally, that limits the solution, like one or two. There in-- isn't just too much space for creativity.
- 13:57
And second of all, not enough focus on open-ended task. We live in the real world with a lot of complexity, uncertainty, and that's not fully captured by the current benchmark.
- 14:10
And also, there isn't enough simulation of the real world. We live in the world. There are other players, like in today's conference room, there are other real people who have a different thought, different gains than you have in your mind.
- 14:23
That's the complexity that's not fully captured. And another thing I want to talk about is the long horizon impact on compute.
- 14:34
We know that GPUs are scarce and precious resources, and in this case of a long-horizon reasoning, you have to be careful about how to optimize your use between training and inference.
- 14:46
And pipeline RL is a quite popular technique nowadays. So basically it's a trade-off between off-policy and the GPU utilization. So traditionally, you let inference run towards the end, and then you start to train the model.
- 15:02
But in the case of long horizon, you have to wait until the inference finish. What the pipeline RL does is that you let the sequence be generated, and you start to train the model while there's still more, uh, sequences being generated.
- 15:17
And you see this create off-policy. But from our experience, normally off-policy up to eight steps is okay. So essentially we made a trade-off between the off-policy and the GPU utilization.
- 15:31
But here comes the issue. As we-- the long horizon indicates, sometimes the inference will take weeks or even more. In that case, inevitably it will goes beyond the constraint of the eight steps of off-policy, so your GPU have just to sit there idle and wait for it to finish.
- 15:50
And if you don't want to wait, as I mentioned before, applying the value model allows you to bootstrap. What it means is that before the end of the episode, you generate expectation.
- 16:02
It's like a dopamine in human brain, and that allows you to train the model. But here's another trade-off. While you utilize the GPU fully, you introduce the value model bias, so there's always a bigger trade-off in those solutions.
- 16:18
And I want to also mention that in the long horizon, infrastructure is important, especially for the environment. And OpenReward was a product, is a platform by General Reasoning. If you're interested, you can check it out, openreward.ai.
- 16:33
So it's a place where it hosts over three hundred and fifty environments and with a single API endpoint, and we use this for our internal RL and also some frontier labs and new labs are using this.
- 16:49
So to summarize both Ross and my speech, it has been a long journey as long horizon indicate.
- 16:58
We as a team have seen the paradigms in AI reasoning on pre-training and agents in the past few years. But looking ahead, what makes us really excited is the long horizon, and it requires us to think, have a new thinking on the algorithm, environments, and compute.
- 17:19
There are a lot of challenges and trade-offs, but we find it's really exciting to take on this journey because, as I mentioned in the very beginning, long horizon is not just an engineering problem, it is a mindset, and if we really are ambitious to solve humanity's biggest problems, this is a journey for everyone, and that's also the
- 17:40
mission for General Reasoning. So if you are interested, follow us, General Reasoning. We're a London-based AI research company. Thank you. [audience applauding] [upbeat music]