# Can AI Agents Beat Wall Street at Predicting Earnings? - Ashwin Paranjape, Samaya AI

Ashwin Paranjape, Samaya AI

[Talk page](https://ai.engineer/talks/SE6POOEjh4I-ai-agents-beat-wall-street-at-predicting) · [Video](https://www.youtube.com/watch?v=SE6POOEjh4I)

## Can AI Agents Beat Wall Street at Predicting Earnings?

Across more than 450 earnings reports, Samaya AI found that one agent forecast revenue more accurately than a bias-corrected analyst consensus. A wafer supplier and a retailer’s tariff refund show how much turns on the step between finding evidence and putting it into a forecast.

### What can an earnings report tell you about a forecast?

[0:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=1s)

Suppose an agent picks a stock and the price goes up. How would you tell whether it got the reasoning right? Ashwin Paranjape, founding AI lead at Samaya AI, came to that question after building tools for financial research. His team could retrieve information from messy customer data, give an agent access to multiple sources within its context window, and evaluate the research it produced. A stock price made for a frustrating next test. Even people following a company often struggle to explain why its stock moved. After a bad call, what would an engineer know to fix?

A colleague suggested predicting quarterly earnings. A company reports figures such as revenue, gross margin, operating income and earnings per share, and gives guidance for the coming quarter. Analysts make their own forecasts. Many sell-side analysts publish theirs, and those estimates are pooled into a **consensus**. When the next report arrives, each figure has a reference point. A result above consensus is a beat. A result below it is a miss.

The reporting cycle gives an engineer three things to work with. The reported figure settles the numerical question. Consensus shows how well professionals did with the information available beforehand. And the earnings call and subsequent commentary discuss what drove the surprise. If an agent called a beat, the engineer can check whether the development it spotted figures in the company’s explanation.

### Stopping the research clock before earnings

[4:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=272s)

Where does the reported number enter the experiment? Take a forecast made seven days, or five trading sessions, before earnings. The model runs inside a harness, the software that gives it tools and context. Those tools can return information available up to the forecast minute. The agent researches the company and issues its estimate. Only on earnings day does the evaluator bring in the reported figure to score that estimate.

Samaya started with its production harness, which already served company documents, prices, consensus estimates and news. The team modified both document retrieval and the structured-data tools to enforce a point-in-time cutoff through an authentication token. That puts the decision about which records to return in software outside the model. Even if the agent asks for a later document, the tool enforces the same cutoff.

The open web posed a practical difficulty. A page available today is hard to reconstruct as it appeared at a particular minute in the past. The team approximated web coverage by adding timestamped news streams that the harness could filter. It also supplied expert guidance on how to use the available context to make a forecast.

### How close did the forecasts come?

[6:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=391s)

A tool cutoff handles one way an agent might know the answer. A historical earnings release could also have been in a model’s training data. Samaya therefore chose reports from July 2026 onward, after the June knowledge cutoff of the newest model it tested. The evaluation covered more than 450 earnings reports and compared seven frontier models.

The first score measured the distance between forecast and actual, scaled by the company’s historical volatility. Published consensus had an error of **1.5** on that scale. But companies beat the published estimate more often than they miss it. An agent could gain some ground simply by allowing for that tendency. Correcting consensus for the bias brought its error down to **1.18**, giving the agents a tougher benchmark.

Where did an agent clear that tougher benchmark? Across the earnings metrics taken together, prediction error tended to fall with newer models. Revenue gave a clearer answer when examined on its own. It required the fewest adjustments of the metrics tested, and **Astra** had significantly lower revenue error than bias-corrected consensus.

### Calling the direction and size of a surprise

[8:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=488s)

- **Hit rate.** If a company reports above consensus, any forecast above consensus gets the direction right. Predicting a slight beat earns the same credit as predicting a large one. To see how much of that success could come from a company’s track record, Samaya looked back over its previous eight quarters. The baseline called whichever outcome, beat or miss, had occurred more often. Newer models began to outperform that rule even on the harder below-consensus calls.

- **Surprise correlation.** When an agent predicts a large beat, does the company tend to deliver a large beat? Do the largest predicted misses also correspond to large actual misses? Correlating predicted and actual departures from consensus uses both direction and magnitude, giving the comparison more statistical power. On this measure, three leading models significantly outperformed the baseline. The experiment could distinguish the leaders from the baseline more confidently than it could distinguish them from one another.

### A wafer supplier changes a revenue forecast

[9:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=570s)

One agent was forecasting earnings for a semiconductor manufacturer that supplied data centers. A clue turned up one step upstream, at a small UK wafer maker that supplied the manufacturer. Information from the wafer maker pointed to stronger revenue in its customer’s data-center business. The agent raised its revenue assumption for that part of the business.

How far did that observation move the final number? The agent carried its revised revenue assumption through the manufacturer’s profit-and-loss statement. Its forecast landed well above both published and bias-corrected consensus, close to the figure the manufacturer ultimately reported. A piece of news about one business had changed the agent’s estimate for a particular line of business at another.

### At Ross Stores, the evidence never reached the estimate

[10:36](https://www.youtube.com/watch?v=SE6POOEjh4I&t=636s)

What happens when the agent finds a clue and leaves it out of the calculation? For Ross Stores, the clue was a tariff refund that another retailer had booked. A similar refund at Ross could lift gross margin. The agent recognized the peer’s refund and chose to leave that possibility out of its Ross forecast. Its gross-margin estimate stayed very close to consensus. Ross reported a much higher margin. Better retrieval alone would have done little here. The agent had already found the item that called for a judgment.

Across the broader set of forecasts, Paranjape also examined calls well above or below consensus. The models tended to outperform when the surprises were larger. Those calls give a researcher a specific decision to investigate in the light of the earnings commentary. What evidence justified moving that far from the analysts’ estimate, and did the company’s result bear it out?

### What changes when the data stops 11 weeks before earnings?

[11:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=707s)

The wafer supplier’s contribution suggests a way to test what the harness adds. How well could the model forecast using only what it had learned during training? Samaya removed external data access. Even a model with a June 2026 training cutoff, which could have learned from documents published shortly before the evaluation period, forecast poorly in that condition.

How much did freshness matter once the model could do research? In a second comparison, the tools supplied information only up to 11 weeks before the earnings call. The agent still had the previous quarter’s results, management guidance and the analyst consensus available then. Losing access to more recent developments cost about as much forecasting performance as the gap between **Sonnet** and **Fable 5.1**. Among the factors Samaya examined, access to up-to-date information mattered most.

### Expert guidance and the cost of hiding consensus

[13:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=789s)

With expert guidance, the agents spent more minutes researching a forecast and consumed more tokens than they did without it. For Astra, guidance also led to a significant reduction in prediction error.

Did the agents need Wall Street’s number as a starting point? The team removed direct access to consensus. Since a news story could still quote the estimate, the document extraction tools also redacted mentions of consensus from news. Older models performed substantially better when they could see the estimate. For Astra, the difference between the visible and hidden conditions was statistically insignificant. Paranjape interpreted that result as evidence that a capable model could reconstruct the consensus view from the remaining information.

### Can an agent learn from the last earnings call?

[14:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=852s)

At the time of the talk, Samaya had made earnings forecasting available for alpha testing and was working with customers to find out whether their internal data improved forecasts. The team was also comparing model reasoning with human reasoning. A decision like the one at Ross gives that question a practical focus. When a peer books a refund, might an analyst and an agent bring different evidence to bear on whether to change the estimate?

Post-training and continual learning were open research directions. For Ross, the agent could revisit the evidence it had, the adjustment it declined to make and the margin the company reported. The question is whether experience with a miss like that can improve the judgment it makes *before* the next report comes out.

For more worked forecasts, Samaya’s accompanying blog post continues the case studies. Frontier Finance is the resource Paranjape points engineers to for evaluating their own financial research harnesses. The two QR codes are on the [closing resource slide](https://www.youtube.com/watch?v=SE6POOEjh4I&t=896s).

## Key takeaways

- A historical forecasting test has two clocks. The forecast minute controls what the tools can retrieve. The model’s training cutoff determines how late the test reports need to be to keep the answers out of its training data.
- Correcting for analysts’ tendency to underestimate results lowered consensus error from 1.5 to 1.18 units of historical volatility. On revenue, Astra beat that stronger benchmark.
- Fresh evidence and the published analyst estimate contributed differently. Restricting data to 11 weeks before earnings materially weakened forecasts. Hiding consensus made no statistically significant difference for Astra.

## Slides

### Can agents forecast markets?

[0:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=28s) [A time-series chart continues into a cloud of possible futures beside the opening question.](https://assets.ai.engineer/sites/aiecode2025/sha256/881df3dc1b3d91bb0c06506b6855325300456fc3b6ee9f3fccc0c4db952df228.webp)

AI Engineer NYC · October 14, 2026. Agents can rewrite compilers [crossed out], solve Navier–Stokes. Why can’t they pick stocks? Ashwin Paranjape. Samaya.

### From Stanford NLP to FrontierFinance

[1:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=67s) [Four milestones trace the work from Stanford NLP and long-context retrieval to a financial agent and its benchmark.](https://assets.ai.engineer/sites/aiecode2025/sha256/dfa0e509def506106d3c9e375c623e06a9127b25f1758d69c644b7e37022f80d.webp)

2019: Stanford NLP. GPT-2 in production with real users. 2023: Lost in the Middle. 2025: Financial Agent harness. 2026: FrontierFinance.

### What can a model forecast?

[2:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=126s) [A flow diagram and a crystal ball contrast mathematical problems with forecasting.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9b859a281104d0d083c9c92578bd177bfaec5e210e6a23dd7bcb27b3841f1f3.webp)

Math: Navier–Stokes and hundreds of open problems. Finance: Can we forecast the future?

### Stock price prediction

[2:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=164s) [Several price movements on a jagged chart are marked with question marks.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9fbc4845c276ce51d1869cb69d318a96893bf572c51bbfe09faa0956408e269.webp)

Stock price prediction? One does not simply start with stock prediction. Arash Alidoust, Samaya.

### Forecasting earnings against analyst consensus

[4:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=263s) [Worked example compares an earnings report with a forecast to define a five-cent surprise.](https://assets.ai.engineer/sites/aiecode2025/sha256/64b3c36c15be399839013c9ae8a62170462820eb58a517decc93532fb93d7b7d.webp)

Earnings prediction. Q1 EPS $0.98. Analyst forecasts: $1.08, $1.04, $1.00. Average forecast $1.04. Q2 EPS $1.09. +$0.05 surprise. Earnings call: explains the drivers.

### Building the environment and harness

[4:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=271s) [Section title for the benchmark environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/20c523ccab8484b96f91a816cd0f2f90bb2a2a93bdaa0271bda773b448d7ee4f.webp)

Setup. Building the environment and harness.

### Prediction day and earnings day

[4:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=298s) [Timeline separates the prediction cutoff from the later earnings report and scoring.](https://assets.ai.engineer/sites/aiecode2025/sha256/c23dc467e3f6e39d8f4c32369f3d5e79c6e34981b42f1cd28bac6b1b932feb75.webp)

Task environment. Samaya’s harness. Point-in-time information: filings & transcripts, price data, news & events, broker estimates. Prediction day: T−5 sessions. Earnings day: T.

### The research harness

[5:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=333s) [Five changes adapt a production research harness to historical forecasting.](https://assets.ai.engineer/sites/aiecode2025/sha256/109fcf1876fb84e4a0f17ff731c96cded07634ff208cc0d3fe47c5f532967e40.webp)

Harness. Start from production harness: includes all real-world data. Enforce point-in-time gate: programmatic with an auth-token. Modify document retrieval and structured data tools. Substitute web access with timestamped news streams. Add expert guidance.

### Predicting earnings surprises

[6:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=389s) [Results section gives the sample size, forecast horizon and number of models tested.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e318463e9aba37a30427c0d976beba90450da1bf8067ad4ed3711fff45682c5.webp)

Results. AI outperforms consensus in predicting earnings surprises. 456 Q2 earnings prints · 5 trading days ahead · 7 frontier models.

### Prediction error by model

[7:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=459s) [Seven bars compare model prediction errors with raw and bias-corrected consensus.](https://assets.ai.engineer/sites/aiecode2025/sha256/1aa1073958c9d15a3c43506077f9011eadb75a9010a9edfc4a9deaf8915fdaf1.webp)

Prediction error. How far off, in units of each company’s typical surprise (σ). Lower is better. Raw consensus 1.57σ. Bias-corrected consensus 1.18σ. GPT-6 Astra 1.17.

### Hit rate

[8:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=486s) [Two bar charts compare the direction of forecasts across all four metrics and for below-consensus calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/fdc285416b1a0871820999795e89dc6d9239cfda594acea655f673e68a3618d1.webp)

Hit rate. Consensus says $1.04 and the print is $1.09. Did the model also call it above consensus? Historical majority: 75.5% overall, 41.5% below consensus. All four metrics. Below-consensus calls.

### Surprise correlation

[8:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=531s) [A seven-model correlation chart marks the 0.45 bias-corrected consensus baseline.](https://assets.ai.engineer/sites/aiecode2025/sha256/3937f94ea8465801275dcc60e187593e54b407a56b643eb0812ffe4e3d799c53.webp)

Surprise correlation. Rank companies by predicted surprise. Are the biggest predicted beats the biggest real ones? Bias-corrected consensus 0.45. Claude Sonnet 5 0.33; Kimi K3 0.38; Gemini 3.8 Flash 0.42; GPT-5.6 Sol 0.46; GPT-6 Astra 0.52; Claude Opus 5.5 0.53; Claude Fable 5.1 0.54.

### When evidence changed the forecast

[9:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=578s) [Supplier evidence in the MACOM example moves the model’s earnings estimate closer to the reported result.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e590dabdd8ced634d5a153f621d2efecbe6b179561df1b10c2343e8d38f1dbc.webp)

Agents win by gathering evidence and adjusting away from consensus. MACOM (MTSI) · Adj. EPS · FQ3 2026. A small UK wafer supplier’s trading update revealed MACOM’s data-center quarter. Consensus $1.352, bias-corrected $1.357, AI forecast $1.39, actual $1.40.

### When a model ignored evidence

[10:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=635s) [The Ross Stores example compares a 29.10% model forecast with a 33.80% reported margin.](https://assets.ai.engineer/sites/aiecode2025/sha256/96ca1446e7929ed2ac26d3514e7d955e33ecb2d4f6ae4846eeebbec9eb6ad676.webp)

And they miss when they see the evidence but don’t act on it. Ross Stores (ROST) · Gross margin · FQ2 2027. A peer had just booked a $120M tariff refund. The AI called it “genuine upside risk” and kept it out of the forecast.

### Performance on larger surprises

[11:19](https://www.youtube.com/watch?v=SE6POOEjh4I&t=679s) [Three groups of bars compare revenue forecast errors by the size of the surprise.](https://assets.ai.engineer/sites/aiecode2025/sha256/1cc797a9debb7e15574eff263833f785df7fcf6af10802ae401cf758bc9e2e70.webp)

Frontier models outperform on bigger surprises. Revenue error (σ) by how far the actual landed from raw consensus. 36 companies below −0.5σ; 183 between −0.5σ and +1σ; 237 above +1σ.

### Separating the data, harness and model

[11:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=706s) [Section title for the experiments separating model, harness and data effects.](https://assets.ai.engineer/sites/aiecode2025/sha256/255bb4bff8d057f4d68bac4f07acbed838c9534c35f564ff268d92b3eb525112.webp)

What drives it. Disentangling the impact of the data, harness and model.

### Live data and forecast accuracy

[12:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=752s) [Two sets of bars compare model correlations using stored knowledge, stale data and recent data.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a7942e22aa4c0adc004c59265d47e61397795d539ab1afdf7556be4048b20c4.webp)

Access to live data is the most important factor in accurate financial predictions. Parametric knowledge only, stale data (T−55), fresh data (T−5). Bias-corrected consensus 0.45. GPT-6 Astra: 0.11, 0.35, 0.51. Claude Fable 5.1: 0.15, 0.40, 0.52.

### Guided research and reasoning

[13:19](https://www.youtube.com/watch?v=SE6POOEjh4I&t=799s) [Paired charts compare research time, context use and model error with and without expert guidance.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc2e0eec7ea392ca50ce8282f19a9b225d018eb653d0df3a7209ea34074e77f5.webp)

Expert guidance strengthens search and reasoning. Research effort. Prediction error (σ). Without guidance, with expert guidance. Claude Fable 5.1: 1.27, 1.25. GPT-6 Astra: 1.22, 1.17.

### Withholding the consensus forecast

[13:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=811s) [Paired bars compare five models with the consensus forecast withheld and provided.](https://assets.ai.engineer/sites/aiecode2025/sha256/1091dbc099613a5e4804e3a07069d876a77b96198e255b31352503582da1f53a.webp)

The best models do well even when the consensus is taken away. Without consensus; with consensus. Kimi K3 0.20 and 0.38; Gemini 3.8 Flash 0.29 and 0.41; GPT-5.6 Sol 0.35 and 0.46; Claude Fable 5.1 0.46 and 0.54; GPT-6 Astra 0.50 and 0.52.

### Next experiments

[14:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=850s) [Final slide gives three next experiments and QR codes for the paper and FrontierFinance.](https://assets.ai.engineer/sites/aiecode2025/sha256/9735c86eb3efd84a07463f29894725ca7a59b6552c4030ea53cd8e6925dda0c0.webp)

Frontier models are outperforming consensus! What’s next? Alpha testing in the product. Research on model reasoning. RL post-training and continual learning. The paper. FrontierFinance.

## Full transcript

[0:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=12s) I'm Ashwin Paranspe. I am uh the

[0:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=14s) I'm Ashwin Paranspe. I am uh the

[0:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=14s) I'm Ashwin Paranspe. I am uh the founding AI lead at Somaya. Uh and today

[0:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=17s) founding AI lead at Somaya. Uh and today

[0:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=17s) founding AI lead at Somaya. Uh and today I'm going to talk about uh you know if

[0:19](https://www.youtube.com/watch?v=SE6POOEjh4I&t=19s) I'm going to talk about uh you know if

[0:19](https://www.youtube.com/watch?v=SE6POOEjh4I&t=19s) I'm going to talk about uh you know if agents can rewrite compilers why can't

[0:21](https://www.youtube.com/watch?v=SE6POOEjh4I&t=21s) agents can rewrite compilers why can't

[0:21](https://www.youtube.com/watch?v=SE6POOEjh4I&t=21s) agents can rewrite compilers why can't they pick stocks? Uh so I give I had

[0:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=24s) they pick stocks? Uh so I give I had

[0:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=24s) they pick stocks? Uh so I give I had this title um couple of months ago but

[0:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=27s) this title um couple of months ago but

[0:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=27s) this title um couple of months ago but AI moves so fast that I now need to

[0:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=29s) AI moves so fast that I now need to

[0:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=29s) AI moves so fast that I now need to change this title to agents can solve

[0:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=31s) change this title to agents can solve

[0:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=31s) change this title to agents can solve Navier Stokes uh and why can't they uh

[0:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=34s) Navier Stokes uh and why can't they uh

[0:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=34s) Navier Stokes uh and why can't they uh pick stocks. So um just a brief

[0:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=37s) pick stocks. So um just a brief

[0:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=37s) pick stocks. So um just a brief background about me uh so I did my PhD

[0:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=40s) background about me uh so I did my PhD

[0:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=40s) background about me uh so I did my PhD from Stanford um one of the highlights

[0:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=43s) from Stanford um one of the highlights

[0:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=43s) from Stanford um one of the highlights was you know fine-tuning GPT2 in 2019

[0:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=46s) was you know fine-tuning GPT2 in 2019

[0:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=46s) was you know fine-tuning GPT2 in 2019 and putting it in production with real

[0:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=48s) and putting it in production with real

[0:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=48s) and putting it in production with real users with tens of thousands of

[0:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=49s) users with tens of thousands of

[0:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=49s) users with tens of thousands of interactions every day. uh when I joined

[0:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=52s) interactions every day. uh when I joined

[0:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=52s) interactions every day. uh when I joined Somaya I was focused on training our

[0:54](https://www.youtube.com/watch?v=SE6POOEjh4I&t=54s) Somaya I was focused on training our

[0:54](https://www.youtube.com/watch?v=SE6POOEjh4I&t=54s) Somaya I was focused on training our retrieval stack uh make working with

[0:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=56s) retrieval stack uh make working with

[0:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=56s) retrieval stack uh make working with long context models in fact uh you might

[0:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=58s) long context models in fact uh you might

[0:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=58s) long context models in fact uh you might have seen a paper called lost in the

[0:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=59s) have seen a paper called lost in the

[1:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=60s) have seen a paper called lost in the middle that was led by my intern Nelson

[1:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=62s) middle that was led by my intern Nelson

[1:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=62s) middle that was led by my intern Nelson when he was interning with us um but

[1:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=64s) when he was interning with us um but

[1:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=64s) when he was interning with us um but yeah so this allowed us to have high

[1:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=66s) yeah so this allowed us to have high

[1:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=66s) yeah so this allowed us to have high quality retrieval over messy customer

[1:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=67s) quality retrieval over messy customer

[1:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=68s) quality retrieval over messy customer data last year uh I kind of changed my

[1:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=71s) data last year uh I kind of changed my

[1:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=71s) data last year uh I kind of changed my focus into designing our financial agent

[1:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=73s) focus into designing our financial agent

[1:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=73s) focus into designing our financial agent harness so this allows contact context

[1:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=75s) harness so this allows contact context

[1:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=75s) harness so this allows contact context efficient unified access to data sources

[1:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=78s) efficient unified access to data sources

[1:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=78s) efficient unified access to data sources and it's rolled out to like 12,000 users

[1:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=80s) and it's rolled out to like 12,000 users

[1:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=80s) and it's rolled out to like 12,000 users users across Morgan Stanley and hundreds

[1:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=82s) users across Morgan Stanley and hundreds

[1:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=82s) users across Morgan Stanley and hundreds of other users in hedge funds and asset

[1:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=84s) of other users in hedge funds and asset

[1:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=84s) of other users in hedge funds and asset management firms. Uh finally this year

[1:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=88s) management firms. Uh finally this year

[1:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=88s) management firms. Uh finally this year we released a benchmark which is the

[1:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=89s) we released a benchmark which is the

[1:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=90s) we released a benchmark which is the largest and the hardest open-source

[1:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=91s) largest and the hardest open-source

[1:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=92s) largest and the hardest open-source benchmark for you know deep research

[1:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=94s) benchmark for you know deep research

[1:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=94s) benchmark for you know deep research financial workflows. Um I led the design

[1:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=97s) financial workflows. Um I led the design

[1:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=97s) financial workflows. Um I led the design of this benchmark a couple of years ago

[1:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=99s) of this benchmark a couple of years ago

[1:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=99s) of this benchmark a couple of years ago when we started collecting it. Please

[1:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=101s) when we started collecting it. Please

[1:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=101s) when we started collecting it. Please check it out if that's of interest to

[1:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=103s) check it out if that's of interest to

[1:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=103s) check it out if that's of interest to you. Um so yeah so I was thinking well

[1:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=105s) you. Um so yeah so I was thinking well

[1:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=105s) you. Um so yeah so I was thinking well you know worked on training the

[1:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=107s) you know worked on training the

[1:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=107s) you know worked on training the retrieval stack the harness eval so

[1:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=109s) retrieval stack the harness eval so

[1:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=109s) retrieval stack the harness eval so what's next right um so in math you have

[1:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=113s) what's next right um so in math you have

[1:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=113s) what's next right um so in math you have like this marquee problem like the

[1:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=115s) like this marquee problem like the

[1:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=115s) like this marquee problem like the millennium price problem can you get a

[1:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=117s) millennium price problem can you get a

[1:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=117s) millennium price problem can you get a solution to navia stokes there are

[1:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=119s) solution to navia stokes there are

[1:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=119s) solution to navia stokes there are hundreds of open math problems like that

[2:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=121s) hundreds of open math problems like that

[2:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=122s) hundreds of open math problems like that so what could something like this look

[2:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=123s) so what could something like this look

[2:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=123s) so what could something like this look for finance right um so like the kind of

[2:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=128s) for finance right um so like the kind of

[2:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=128s) for finance right um so like the kind of zoomed out answer is can we forecast the

[2:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=131s) zoomed out answer is can we forecast the

[2:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=131s) zoomed out answer is can we forecast the future can we forecast it better than

[2:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=133s) future can we forecast it better than

[2:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=133s) future can we forecast it better than what we do today and that's really like

[2:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=135s) what we do today and that's really like

[2:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=135s) what we do today and that's really like the the highest level challenge we can

[2:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=137s) the the highest level challenge we can

[2:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=137s) the the highest level challenge we can focus on here. So you know in my naivity

[2:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=140s) focus on here. So you know in my naivity

[2:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=140s) focus on here. So you know in my naivity I went around asking people well what if

[2:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=143s) I went around asking people well what if

[2:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=143s) I went around asking people well what if you try to predict stocks. Um so just a

[2:26](https://www.youtube.com/watch?v=SE6POOEjh4I&t=146s) you try to predict stocks. Um so just a

[2:26](https://www.youtube.com/watch?v=SE6POOEjh4I&t=146s) you try to predict stocks. Um so just a show of hands here like how many people

[2:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=148s) show of hands here like how many people

[2:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=148s) show of hands here like how many people try to predict stocks or are adjacent to

[2:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=151s) try to predict stocks or are adjacent to

[2:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=151s) try to predict stocks or are adjacent to it in this audience?

[2:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=154s) it in this audience?

[2:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=154s) it in this audience? Wow that's like over over half of us. Um

[2:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=157s) Wow that's like over over half of us. Um

[2:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=157s) Wow that's like over over half of us. Um and so um so I was like okay well you

[2:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=159s) and so um so I was like okay well you

[2:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=159s) and so um so I was like okay well you know how hard could it be like

[2:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=161s) know how hard could it be like

[2:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=161s) know how hard could it be like everybody's trying to do this. So uh I

[2:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=164s) everybody's trying to do this. So uh I

[2:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=164s) everybody's trying to do this. So uh I went to Arash who's a colleague here at

[2:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=166s) went to Arash who's a colleague here at

[2:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=166s) went to Arash who's a colleague here at Somaya. He was previously at Bridgewater

[2:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=169s) Somaya. He was previously at Bridgewater

[2:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=169s) Somaya. He was previously at Bridgewater and Varition and he says like you know

[2:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=171s) and Varition and he says like you know

[2:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=171s) and Varition and he says like you know you just don't simply start with stock

[2:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=172s) you just don't simply start with stock

[2:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=172s) you just don't simply start with stock prediction. It's very noisy. It um

[2:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=175s) prediction. It's very noisy. It um

[2:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=175s) prediction. It's very noisy. It um doesn't tell you why a certain model is

[2:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=178s) doesn't tell you why a certain model is

[2:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=178s) doesn't tell you why a certain model is behaving in certain ways. It doesn't

[2:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=179s) behaving in certain ways. It doesn't

[2:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=179s) behaving in certain ways. It doesn't tell you if you know the reasoning is

[3:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=181s) tell you if you know the reasoning is

[3:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=181s) tell you if you know the reasoning is right or not. Nobody really knows why

[3:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=183s) right or not. Nobody really knows why

[3:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=183s) right or not. Nobody really knows why stock moves a lot of times. So then u by

[3:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=187s) stock moves a lot of times. So then u by

[3:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=187s) stock moves a lot of times. So then u by the way he's sitting here uh close

[3:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=188s) the way he's sitting here uh close

[3:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=188s) the way he's sitting here uh close collaborator

[3:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=190s) collaborator

[3:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=190s) collaborator [laughter]

[3:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=192s) [laughter]

[3:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=192s) [laughter] come with us afterwards. So um yeah so

[3:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=196s) come with us afterwards. So um yeah so

[3:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=196s) come with us afterwards. So um yeah so he suggested let's instead do earnings

[3:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=198s) he suggested let's instead do earnings

[3:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=198s) he suggested let's instead do earnings prediction. So what is earnings right?

[3:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=200s) prediction. So what is earnings right?

[3:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=200s) prediction. So what is earnings right? So every quarter a company uh comes out

[3:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=203s) So every quarter a company uh comes out

[3:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=203s) So every quarter a company uh comes out and reports its earnings. It gives

[3:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=205s) and reports its earnings. It gives

[3:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=205s) and reports its earnings. It gives revenue, gross margin, operating income,

[3:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=208s) revenue, gross margin, operating income,

[3:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=208s) revenue, gross margin, operating income, uh EPS, a bunch of other metrics and

[3:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=211s) uh EPS, a bunch of other metrics and

[3:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=211s) uh EPS, a bunch of other metrics and then it also gives guidance for the next

[3:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=212s) then it also gives guidance for the next

[3:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=212s) then it also gives guidance for the next quarter. Um analysts uh across buy side

[3:36](https://www.youtube.com/watch?v=SE6POOEjh4I&t=216s) quarter. Um analysts uh across buy side

[3:36](https://www.youtube.com/watch?v=SE6POOEjh4I&t=216s) quarter. Um analysts uh across buy side and sell side try to predict these

[3:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=217s) and sell side try to predict these

[3:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=217s) and sell side try to predict these numbers in the coming quarter. Uh a lot

[3:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=220s) numbers in the coming quarter. Uh a lot

[3:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=220s) numbers in the coming quarter. Uh a lot of the sellside analysts publish this

[3:42](https://www.youtube.com/watch?v=SE6POOEjh4I&t=222s) of the sellside analysts publish this

[3:42](https://www.youtube.com/watch?v=SE6POOEjh4I&t=222s) of the sellside analysts publish this right. So it gets pulled as what is

[3:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=224s) right. So it gets pulled as what is

[3:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=224s) right. So it gets pulled as what is called a consensus estimate. And then

[3:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=226s) called a consensus estimate. And then

[3:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=226s) called a consensus estimate. And then the next quarter the company reports.

[3:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=228s) the next quarter the company reports.

[3:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=228s) the next quarter the company reports. Now if it's above the consensus it's a

[3:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=230s) Now if it's above the consensus it's a

[3:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=230s) Now if it's above the consensus it's a beat and it's a surprise. If it's below

[3:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=232s) beat and it's a surprise. If it's below

[3:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=232s) beat and it's a surprise. If it's below the consensus it's a miss. Uh and then

[3:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=235s) the consensus it's a miss. Uh and then

[3:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=235s) the consensus it's a miss. Uh and then during the call and after the call in

[3:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=236s) during the call and after the call in

[3:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=236s) during the call and after the call in the commentary they also talk about why

[3:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=238s) the commentary they also talk about why

[3:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=238s) the commentary they also talk about why these uh like you know surprises

[4:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=240s) these uh like you know surprises

[4:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=240s) these uh like you know surprises happened. So this is from a from a AI

[4:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=243s) happened. So this is from a from a AI

[4:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=243s) happened. So this is from a from a AI and ML perspective this is a great task.

[4:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=246s) and ML perspective this is a great task.

[4:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=246s) and ML perspective this is a great task. It has qual quantitative ground truth.

[4:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=249s) It has qual quantitative ground truth.

[4:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=249s) It has qual quantitative ground truth. You know what exactly happened

[4:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=250s) You know what exactly happened

[4:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=250s) You know what exactly happened afterwards. you have an expert human

[4:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=252s) afterwards. you have an expert human

[4:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=252s) afterwards. you have an expert human baseline to compare against and you also

[4:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=255s) baseline to compare against and you also

[4:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=255s) baseline to compare against and you also have the qualitative ground truth that

[4:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=257s) have the qualitative ground truth that

[4:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=257s) have the qualitative ground truth that tells you you know why the agent was

[4:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=260s) tells you you know why the agent was

[4:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=260s) tells you you know why the agent was right if it was right and so the then

[4:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=263s) right if it was right and so the then

[4:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=263s) right if it was right and so the then the question is well can an agent do

[4:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=265s) the question is well can an agent do

[4:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=265s) the question is well can an agent do this task can they make good predictions

[4:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=268s) this task can they make good predictions

[4:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=268s) this task can they make good predictions and this is this is going to be the

[4:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=269s) and this is this is going to be the

[4:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=269s) and this is this is going to be the focus of today's talk

[4:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=272s) focus of today's talk

[4:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=272s) focus of today's talk so to be able to do this we had to work

[4:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=275s) so to be able to do this we had to work

[4:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=275s) so to be able to do this we had to work on building an environment and a harness

[4:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=277s) on building an environment and a harness

[4:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=277s) on building an environment and a harness so just to give you background on

[4:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=279s) so just to give you background on

[4:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=279s) so just to give you background on environments whenever you have these

[4:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=280s) environments whenever you have these

[4:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=280s) environments whenever you have these agent intake models they need to be in

[4:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=283s) agent intake models they need to be in

[4:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=283s) agent intake models they need to be in an environment where you know they are

[4:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=284s) an environment where you know they are

[4:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=284s) an environment where you know they are given the right data they are given a

[4:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=285s) given the right data they are given a

[4:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=285s) given the right data they are given a task and then you can evaluate it at the

[4:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=287s) task and then you can evaluate it at the

[4:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=287s) task and then you can evaluate it at the end and you might have heard it in the

[4:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=289s) end and you might have heard it in the

[4:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=289s) end and you might have heard it in the context of training so RL training but

[4:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=292s) context of training so RL training but

[4:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=292s) context of training so RL training but you need the same environment for

[4:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=293s) you need the same environment for

[4:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=293s) you need the same environment for evaluation as well and then uh that

[4:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=296s) evaluation as well and then uh that

[4:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=296s) evaluation as well and then uh that environment needs to provide a harness

[4:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=298s) environment needs to provide a harness

[4:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=298s) environment needs to provide a harness to the agent so here's how the

[5:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=300s) to the agent so here's how the

[5:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=300s) to the agent so here's how the environment schematically looks like so

[5:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=302s) environment schematically looks like so

[5:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=302s) environment schematically looks like so you have an AI model that's sitting

[5:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=305s) you have an AI model that's sitting

[5:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=305s) you have an AI model that's sitting inside a a harness so we provided the

[5:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=308s) inside a a harness so we provided the

[5:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=308s) inside a a harness so we provided the harness this harness we constructed such

[5:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=310s) harness this harness we constructed such

[5:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=310s) harness this harness we constructed such that it brings point in time

[5:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=312s) that it brings point in time

[5:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=312s) that it brings point in time information. Um so if we are trying to

[5:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=315s) information. Um so if we are trying to

[5:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=315s) information. Um so if we are trying to predict say 7 days or five sessions

[5:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=318s) predict say 7 days or five sessions

[5:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=318s) predict say 7 days or five sessions before earnings the context only

[5:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=320s) before earnings the context only

[5:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=320s) before earnings the context only includes information up to that point up

[5:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=322s) includes information up to that point up

[5:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=322s) includes information up to that point up to this uh minute and okay so with this

[5:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=325s) to this uh minute and okay so with this

[5:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=325s) to this uh minute and okay so with this harness the AI agent researches and

[5:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=327s) harness the AI agent researches and

[5:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=327s) harness the AI agent researches and makes a prediction and come the earnings

[5:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=329s) makes a prediction and come the earnings

[5:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=329s) makes a prediction and come the earnings day we compare the differences and then

[5:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=331s) day we compare the differences and then

[5:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=331s) day we compare the differences and then see how good its prediction was.

[5:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=334s) see how good its prediction was.

[5:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=334s) see how good its prediction was. Now to build this harness what we did

[5:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=335s) Now to build this harness what we did

[5:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=335s) Now to build this harness what we did was we started from Somaya's production

[5:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=337s) was we started from Somaya's production

[5:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=338s) was we started from Somaya's production grade harness. So this includes all the

[5:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=339s) grade harness. So this includes all the

[5:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=340s) grade harness. So this includes all the real world data that we provide through

[5:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=341s) real world data that we provide through

[5:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=341s) real world data that we provide through our system. Uh it includes uh company

[5:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=344s) our system. Uh it includes uh company

[5:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=344s) our system. Uh it includes uh company documents, prices, uh consensus also

[5:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=347s) documents, prices, uh consensus also

[5:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=347s) documents, prices, uh consensus also news. Then we have an uh point in time

[5:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=351s) news. Then we have an uh point in time

[5:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=351s) news. Then we have an uh point in time gate. So it's a programmatic gate that

[5:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=353s) gate. So it's a programmatic gate that

[5:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=353s) gate. So it's a programmatic gate that enforces no access to future information

[5:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=356s) enforces no access to future information

[5:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=356s) enforces no access to future information and it's enforced with an authentication

[5:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=358s) and it's enforced with an authentication

[5:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=358s) and it's enforced with an authentication token. So there is no way for the agent

[6:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=360s) token. So there is no way for the agent

[6:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=360s) token. So there is no way for the agent to hack around this. uh we modify all of

[6:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=363s) to hack around this. uh we modify all of

[6:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=363s) to hack around this. uh we modify all of our document retrieval and structured

[6:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=365s) our document retrieval and structured

[6:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=365s) our document retrieval and structured data tools to support this point in time

[6:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=367s) data tools to support this point in time

[6:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=367s) data tools to support this point in time gate and web is one of the trickier ones

[6:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=370s) gate and web is one of the trickier ones

[6:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=370s) gate and web is one of the trickier ones where it's very hard to have this gate.

[6:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=372s) where it's very hard to have this gate.

[6:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=372s) where it's very hard to have this gate. So instead we get a reasonable proxy by

[6:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=376s) So instead we get a reasonable proxy by

[6:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=376s) So instead we get a reasonable proxy by adding more new streams and making sure

[6:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=378s) adding more new streams and making sure

[6:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=378s) adding more new streams and making sure that they are timestamped. So uh finally

[6:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=380s) that they are timestamped. So uh finally

[6:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=380s) that they are timestamped. So uh finally we add some expert guidance that helps

[6:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=383s) we add some expert guidance that helps

[6:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=383s) we add some expert guidance that helps with um making better use of the context

[6:26](https://www.youtube.com/watch?v=SE6POOEjh4I&t=386s) with um making better use of the context

[6:26](https://www.youtube.com/watch?v=SE6POOEjh4I&t=386s) with um making better use of the context and also helps with the predictive task.

[6:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=391s) and also helps with the predictive task.

[6:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=391s) and also helps with the predictive task. So this is the headline. So we found

[6:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=393s) So this is the headline. So we found

[6:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=393s) So this is the headline. So we found that AI is able starting to outperform

[6:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=395s) that AI is able starting to outperform

[6:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=395s) that AI is able starting to outperform consensus in predicting earning

[6:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=397s) consensus in predicting earning

[6:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=397s) consensus in predicting earning surprises. Uh here we looked at more

[6:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=400s) surprises. Uh here we looked at more

[6:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=400s) surprises. Uh here we looked at more than 450 earning sprints. This had to be

[6:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=403s) than 450 earning sprints. This had to be

[6:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=403s) than 450 earning sprints. This had to be from July onwards because the latest

[6:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=406s) from July onwards because the latest

[6:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=406s) from July onwards because the latest model that we were testing uh was Fable

[6:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=408s) model that we were testing uh was Fable

[6:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=408s) model that we were testing uh was Fable 5.1 which had a knowledge cut off of

[6:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=410s) 5.1 which had a knowledge cut off of

[6:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=410s) 5.1 which had a knowledge cut off of June. Um so we looked at seven frontier

[6:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=413s) June. Um so we looked at seven frontier

[6:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=413s) June. Um so we looked at seven frontier models to compare and here are some of

[6:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=415s) models to compare and here are some of

[6:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=415s) models to compare and here are some of the results. Um so the first thing we

[6:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=418s) the results. Um so the first thing we

[6:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=418s) the results. Um so the first thing we looked at was prediction error. So this

[7:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=420s) looked at was prediction error. So this

[7:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=420s) looked at was prediction error. So this is um when your agent is making a

[7:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=423s) is um when your agent is making a

[7:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=423s) is um when your agent is making a prediction how far is it from the actual

[7:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=425s) prediction how far is it from the actual

[7:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=425s) prediction how far is it from the actual normalized by the uh company's

[7:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=428s) normalized by the uh company's

[7:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=428s) normalized by the uh company's historical volatility. Uh as a

[7:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=431s) historical volatility. Uh as a

[7:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=431s) historical volatility. Uh as a comparison point we looked at consensus.

[7:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=433s) comparison point we looked at consensus.

[7:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=434s) comparison point we looked at consensus. It had an error of 1.5

[7:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=436s) It had an error of 1.5

[7:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=436s) It had an error of 1.5 uh times historical volatility. But we

[7:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=440s) uh times historical volatility. But we

[7:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=440s) uh times historical volatility. But we also know that a lot of uh consensus is

[7:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=444s) also know that a lot of uh consensus is

[7:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=444s) also know that a lot of uh consensus is under reportported and uh at earnings

[7:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=447s) under reportported and uh at earnings

[7:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=447s) under reportported and uh at earnings you know the companies outperform more

[7:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=449s) you know the companies outperform more

[7:29](https://www.youtube.com/watch?v=SE6POOEjh4I&t=449s) you know the companies outperform more often than not. So we corrected for that

[7:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=451s) often than not. So we corrected for that

[7:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=451s) often than not. So we corrected for that bias and uh we have a much better bias

[7:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=454s) bias and uh we have a much better bias

[7:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=454s) bias and uh we have a much better bias corrected error which is 1.18

[7:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=457s) corrected error which is 1.18

[7:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=458s) corrected error which is 1.18 and here's how the progression of models

[7:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=460s) and here's how the progression of models

[7:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=460s) and here's how the progression of models looks like. So you start from cla set

[7:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=463s) looks like. So you start from cla set

[7:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=463s) looks like. So you start from cla set which is you know Kim K3 some of the

[7:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=465s) which is you know Kim K3 some of the

[7:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=465s) which is you know Kim K3 some of the weaker models but then as you go to the

[7:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=467s) weaker models but then as you go to the

[7:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=467s) weaker models but then as you go to the latest models opus 5.5 fable 5.1 GP6

[7:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=471s) latest models opus 5.5 fable 5.1 GP6

[7:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=471s) latest models opus 5.5 fable 5.1 GP6 Astra you start seeing the error go down

[7:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=473s) Astra you start seeing the error go down

[7:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=473s) Astra you start seeing the error go down uh and in fact this is the graph with

[7:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=476s) uh and in fact this is the graph with

[7:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=476s) uh and in fact this is the graph with all the metrics together but if you look

[7:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=477s) all the metrics together but if you look

[7:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=477s) all the metrics together but if you look at revenue which has the least amount of

[7:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=479s) at revenue which has the least amount of

[7:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=479s) at revenue which has the least amount of adjustments you actually start seeing uh

[8:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=482s) adjustments you actually start seeing uh

[8:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=482s) adjustments you actually start seeing uh GBD6 Ara is significantly better than

[8:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=485s) GBD6 Ara is significantly better than

[8:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=485s) GBD6 Ara is significantly better than even the bias corrected consensus.

[8:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=488s) even the bias corrected consensus.

[8:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=488s) even the bias corrected consensus. We also looked at hit rate. So uh in

[8:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=491s) We also looked at hit rate. So uh in

[8:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=491s) We also looked at hit rate. So uh in simple terms, let's say if the consensus

[8:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=492s) simple terms, let's say if the consensus

[8:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=492s) simple terms, let's say if the consensus is $114,

[8:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=495s) is $114,

[8:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=495s) is $114, the print is $19. So did the model call

[8:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=498s) the print is $19. So did the model call

[8:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=498s) the print is $19. So did the model call it above consensus? If so, that's a hit.

[8:21](https://www.youtube.com/watch?v=SE6POOEjh4I&t=501s) it above consensus? If so, that's a hit.

[8:21](https://www.youtube.com/watch?v=SE6POOEjh4I&t=501s) it above consensus? If so, that's a hit. Otherwise, it's a miss. So here also we

[8:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=503s) Otherwise, it's a miss. So here also we

[8:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=503s) Otherwise, it's a miss. So here also we see the latest models are starting to do

[8:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=505s) see the latest models are starting to do

[8:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=505s) see the latest models are starting to do better. And interestingly, uh looking at

[8:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=508s) better. And interestingly, uh looking at

[8:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=508s) better. And interestingly, uh looking at just the below consensus calls, which

[8:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=510s) just the below consensus calls, which

[8:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=510s) just the below consensus calls, which tend to be harder typically to make, uh

[8:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=512s) tend to be harder typically to make, uh

[8:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=512s) tend to be harder typically to make, uh we see that newer models are starting to

[8:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=515s) we see that newer models are starting to

[8:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=515s) we see that newer models are starting to outperform uh historical majority

[8:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=517s) outperform uh historical majority

[8:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=517s) outperform uh historical majority baseline. So just to talk about the

[8:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=519s) baseline. So just to talk about the

[8:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=519s) baseline. So just to talk about the baseline we looked at last eight

[8:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=520s) baseline we looked at last eight

[8:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=520s) baseline we looked at last eight quarters and saw what was the majority

[8:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=523s) quarters and saw what was the majority

[8:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=523s) quarters and saw what was the majority did it on the whole beat or miss

[8:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=525s) did it on the whole beat or miss

[8:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=525s) did it on the whole beat or miss consensus in the last eight quarters. Um

[8:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=528s) consensus in the last eight quarters. Um

[8:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=528s) consensus in the last eight quarters. Um so yeah so Astra again doing

[8:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=530s) so yeah so Astra again doing

[8:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=530s) so yeah so Astra again doing significantly better and then uh we also

[8:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=533s) significantly better and then uh we also

[8:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=533s) significantly better and then uh we also have a new metric which is you know um

[8:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=536s) have a new metric which is you know um

[8:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=536s) have a new metric which is you know um surprise correlation. So was a big uh

[8:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=539s) surprise correlation. So was a big uh

[8:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=539s) surprise correlation. So was a big uh predicted beat also a big actual beat

[9:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=543s) predicted beat also a big actual beat

[9:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=543s) predicted beat also a big actual beat versus a big um predicted miss a big

[9:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=546s) versus a big um predicted miss a big

[9:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=546s) versus a big um predicted miss a big actual miss. So that allows us to

[9:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=549s) actual miss. So that allows us to

[9:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=549s) actual miss. So that allows us to combine both the um magnitude as well as

[9:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=552s) combine both the um magnitude as well as

[9:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=552s) combine both the um magnitude as well as the direction together and gives us more

[9:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=554s) the direction together and gives us more

[9:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=554s) the direction together and gives us more statistical power uh over all of these

[9:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=557s) statistical power uh over all of these

[9:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=557s) statistical power uh over all of these models. And so here we see again Astra,

[9:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=560s) models. And so here we see again Astra,

[9:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=560s) models. And so here we see again Astra, Opus and Fable are significantly

[9:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=562s) Opus and Fable are significantly

[9:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=562s) Opus and Fable are significantly outperforming the baseline. And uh there

[9:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=565s) outperforming the baseline. And uh there

[9:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=565s) outperforming the baseline. And uh there is uh you know with they are within

[9:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=568s) is uh you know with they are within

[9:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=568s) is uh you know with they are within statistical significance of each other.

[9:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=570s) statistical significance of each other.

[9:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=570s) statistical significance of each other. Okay. So we looked at a bunch of

[9:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=571s) Okay. So we looked at a bunch of

[9:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=571s) Okay. So we looked at a bunch of numbers. You know the numbers tell one

[9:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=573s) numbers. You know the numbers tell one

[9:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=573s) numbers. You know the numbers tell one part of the story. But we also want to

[9:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=575s) part of the story. But we also want to

[9:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=575s) part of the story. But we also want to know what's really happening uh under

[9:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=577s) know what's really happening uh under

[9:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=577s) know what's really happening uh under the hood, right? So um here's here's an

[9:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=581s) the hood, right? So um here's here's an

[9:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=581s) the hood, right? So um here's here's an example. Um what we find is that agents

[9:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=585s) example. Um what we find is that agents

[9:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=585s) example. Um what we find is that agents win when they are gathering evidence and

[9:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=587s) win when they are gathering evidence and

[9:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=587s) win when they are gathering evidence and they are being a they are able to adjust

[9:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=589s) they are being a they are able to adjust

[9:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=589s) they are being a they are able to adjust away from consensus. So in this example

[9:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=592s) away from consensus. So in this example

[9:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=592s) away from consensus. So in this example we have MACM uh which is a

[9:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=595s) we have MACM uh which is a

[9:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=595s) we have MACM uh which is a semi-manufacturer that supplies data

[9:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=597s) semi-manufacturer that supplies data

[9:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=597s) semi-manufacturer that supplies data centers and then the agent found one of

[9:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=599s) centers and then the agent found one of

[9:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=599s) centers and then the agent found one of its suppliers a small UK wafer uh

[10:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=602s) its suppliers a small UK wafer uh

[10:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=602s) its suppliers a small UK wafer uh manufacturer uh it uh it revealed sort

[10:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=607s) manufacturer uh it uh it revealed sort

[10:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=607s) manufacturer uh it uh it revealed sort of MACM's data center improvements like

[10:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=610s) of MACM's data center improvements like

[10:10](https://www.youtube.com/watch?v=SE6POOEjh4I&t=610s) of MACM's data center improvements like in revenue and it adjusted for it uh

[10:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=613s) in revenue and it adjusted for it uh

[10:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=613s) in revenue and it adjusted for it uh went all the way down to the P&L and

[10:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=616s) went all the way down to the P&L and

[10:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=616s) went all the way down to the P&L and found uh and predicted a forecast that

[10:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=618s) found uh and predicted a forecast that

[10:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=618s) found uh and predicted a forecast that was significantly higher than consensus

[10:21](https://www.youtube.com/watch?v=SE6POOEjh4I&t=621s) was significantly higher than consensus

[10:21](https://www.youtube.com/watch?v=SE6POOEjh4I&t=621s) was significantly higher than consensus or bias corrected consensus and very

[10:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=623s) or bias corrected consensus and very

[10:23](https://www.youtube.com/watch?v=SE6POOEjh4I&t=623s) or bias corrected consensus and very close to actual. So this is the kind of

[10:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=625s) close to actual. So this is the kind of

[10:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=625s) close to actual. So this is the kind of reasoning uh these models are doing

[10:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=627s) reasoning uh these models are doing

[10:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=627s) reasoning uh these models are doing where they're able to find data uh that

[10:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=630s) where they're able to find data uh that

[10:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=630s) where they're able to find data uh that is very timely uh over a large scale and

[10:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=632s) is very timely uh over a large scale and

[10:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=632s) is very timely uh over a large scale and then are able to adjust for it. Uh on

[10:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=635s) then are able to adjust for it. Uh on

[10:36](https://www.youtube.com/watch?v=SE6POOEjh4I&t=636s) then are able to adjust for it. Uh on the other hand uh we also find that they

[10:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=638s) the other hand uh we also find that they

[10:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=638s) the other hand uh we also find that they miss when they see the evidence but they

[10:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=640s) miss when they see the evidence but they

[10:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=640s) miss when they see the evidence but they don't have enough conviction to act on

[10:42](https://www.youtube.com/watch?v=SE6POOEjh4I&t=642s) don't have enough conviction to act on

[10:42](https://www.youtube.com/watch?v=SE6POOEjh4I&t=642s) don't have enough conviction to act on it. So this is Ross stores uh the dress

[10:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=645s) it. So this is Ross stores uh the dress

[10:45](https://www.youtube.com/watch?v=SE6POOEjh4I&t=645s) it. So this is Ross stores uh the dress for less Ross and um here the agent

[10:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=649s) for less Ross and um here the agent

[10:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=649s) for less Ross and um here the agent found that one of its peers had booked a

[10:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=651s) found that one of its peers had booked a

[10:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=651s) found that one of its peers had booked a tariff refund but then it decided to

[10:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=653s) tariff refund but then it decided to

[10:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=653s) tariff refund but then it decided to leave it out of its um prediction for

[10:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=658s) leave it out of its um prediction for

[10:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=658s) leave it out of its um prediction for um gross margin and then the prediction

[11:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=661s) um gross margin and then the prediction

[11:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=661s) um gross margin and then the prediction uh like very was very close to the

[11:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=662s) uh like very was very close to the

[11:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=662s) uh like very was very close to the consensus but the actual came out much

[11:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=664s) consensus but the actual came out much

[11:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=664s) consensus but the actual came out much higher and so we see sort of them

[11:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=667s) higher and so we see sort of them

[11:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=667s) higher and so we see sort of them missing when they don't work with

[11:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=669s) missing when they don't work with

[11:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=669s) missing when they don't work with conviction. These are two examples.

[11:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=671s) conviction. These are two examples.

[11:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=671s) conviction. These are two examples. There are more examples on the online

[11:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=673s) There are more examples on the online

[11:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=673s) There are more examples on the online blog post. U but yeah, this is just to

[11:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=675s) blog post. U but yeah, this is just to

[11:15](https://www.youtube.com/watch?v=SE6POOEjh4I&t=675s) blog post. U but yeah, this is just to give a flavor of sort of what's

[11:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=676s) give a flavor of sort of what's

[11:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=676s) give a flavor of sort of what's happening behind the scenes. U okay so

[11:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=680s) happening behind the scenes. U okay so

[11:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=680s) happening behind the scenes. U okay so uh here's the last interesting thing

[11:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=682s) uh here's the last interesting thing

[11:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=682s) uh here's the last interesting thing before we kind of jump into some of the

[11:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=684s) before we kind of jump into some of the

[11:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=684s) before we kind of jump into some of the uplations. So uh if you look at you know

[11:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=688s) uplations. So uh if you look at you know

[11:28](https://www.youtube.com/watch?v=SE6POOEjh4I&t=688s) uplations. So uh if you look at you know these predictions which are

[11:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=690s) these predictions which are

[11:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=690s) these predictions which are significantly above or below consensus

[11:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=692s) significantly above or below consensus

[11:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=692s) significantly above or below consensus we find that the models tend to

[11:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=694s) we find that the models tend to

[11:34](https://www.youtube.com/watch?v=SE6POOEjh4I&t=694s) we find that the models tend to outperform when the surprises are bigger

[11:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=697s) outperform when the surprises are bigger

[11:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=697s) outperform when the surprises are bigger which is also very interesting. like you

[11:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=698s) which is also very interesting. like you

[11:38](https://www.youtube.com/watch?v=SE6POOEjh4I&t=698s) which is also very interesting. like you can look at the things that these models

[11:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=700s) can look at the things that these models

[11:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=700s) can look at the things that these models tend to uh have on the extremes and then

[11:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=703s) tend to uh have on the extremes and then

[11:43](https://www.youtube.com/watch?v=SE6POOEjh4I&t=703s) tend to uh have on the extremes and then learn sort of what they are doing there.

[11:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=707s) learn sort of what they are doing there.

[11:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=707s) learn sort of what they are doing there. Okay. So we looked at how a model can

[11:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=709s) Okay. So we looked at how a model can

[11:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=709s) Okay. So we looked at how a model can change our predictions. But what about

[11:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=711s) change our predictions. But what about

[11:51](https://www.youtube.com/watch?v=SE6POOEjh4I&t=711s) change our predictions. But what about the data and the hardness that goes into

[11:54](https://www.youtube.com/watch?v=SE6POOEjh4I&t=714s) the data and the hardness that goes into

[11:54](https://www.youtube.com/watch?v=SE6POOEjh4I&t=714s) the data and the hardness that goes into it, right?

[11:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=715s) it, right?

[11:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=715s) it, right? So um we found that access to life data

[11:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=718s) So um we found that access to life data

[11:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=718s) So um we found that access to life data is the most important factor in accurate

[12:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=721s) is the most important factor in accurate

[12:01](https://www.youtube.com/watch?v=SE6POOEjh4I&t=721s) is the most important factor in accurate financial predictions. So uh we ran two

[12:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=724s) financial predictions. So uh we ran two

[12:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=724s) financial predictions. So uh we ran two comparisons. one is with just the

[12:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=727s) comparisons. one is with just the

[12:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=727s) comparisons. one is with just the parametric knowledge. So no data access

[12:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=729s) parametric knowledge. So no data access

[12:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=729s) parametric knowledge. So no data access just the model itself. And this gives us

[12:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=731s) just the model itself. And this gives us

[12:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=732s) just the model itself. And this gives us a good comparison point because uh Fable

[12:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=734s) a good comparison point because uh Fable

[12:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=734s) a good comparison point because uh Fable was trained until uh June 2026. And so

[12:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=738s) was trained until uh June 2026. And so

[12:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=738s) was trained until uh June 2026. And so technically it should have memorized all

[12:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=740s) technically it should have memorized all

[12:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=740s) technically it should have memorized all the documents that came out before that.

[12:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=742s) the documents that came out before that.

[12:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=742s) the documents that came out before that. But even uh with all of that in its

[12:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=745s) But even uh with all of that in its

[12:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=745s) But even uh with all of that in its pre-training, it's not able to be like

[12:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=747s) pre-training, it's not able to be like

[12:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=747s) pre-training, it's not able to be like it's not any good at all. Okay. So

[12:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=750s) it's not any good at all. Okay. So

[12:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=750s) it's not any good at all. Okay. So that's the parametric version. What

[12:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=752s) that's the parametric version. What

[12:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=752s) that's the parametric version. What about if we give it stale data? So if

[12:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=755s) about if we give it stale data? So if

[12:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=755s) about if we give it stale data? So if you give it data that is 11 weeks before

[12:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=757s) you give it data that is 11 weeks before

[12:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=757s) you give it data that is 11 weeks before the earnings call uh it still has the

[12:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=760s) the earnings call uh it still has the

[12:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=760s) the earnings call uh it still has the previous earnings result and it's like

[12:42](https://www.youtube.com/watch?v=SE6POOEjh4I&t=762s) previous earnings result and it's like

[12:42](https://www.youtube.com/watch?v=SE6POOEjh4I&t=762s) previous earnings result and it's like you know the guidance and other analyst

[12:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=764s) you know the guidance and other analyst

[12:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=764s) you know the guidance and other analyst consensus but it's around 12 to 15

[12:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=768s) consensus but it's around 12 to 15

[12:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=768s) consensus but it's around 12 to 15 points lower and to put this in context

[12:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=770s) points lower and to put this in context

[12:50](https://www.youtube.com/watch?v=SE6POOEjh4I&t=770s) points lower and to put this in context this is the same gap you would see

[12:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=772s) this is the same gap you would see

[12:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=772s) this is the same gap you would see between sonnet and fable 5.1 in terms of

[12:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=775s) between sonnet and fable 5.1 in terms of

[12:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=775s) between sonnet and fable 5.1 in terms of the quality you would get out of these

[12:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=776s) the quality you would get out of these

[12:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=776s) the quality you would get out of these two models. Uh yeah so latest data is

[13:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=780s) two models. Uh yeah so latest data is

[13:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=780s) two models. Uh yeah so latest data is important also another proof point that

[13:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=782s) important also another proof point that

[13:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=782s) important also another proof point that they are actually you know finding

[13:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=783s) they are actually you know finding

[13:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=783s) they are actually you know finding information and adjusting for it in the

[13:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=786s) information and adjusting for it in the

[13:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=786s) information and adjusting for it in the uh for the latest information.

[13:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=789s) uh for the latest information.

[13:09](https://www.youtube.com/watch?v=SE6POOEjh4I&t=789s) uh for the latest information. Then uh we looked at the effect of

[13:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=791s) Then uh we looked at the effect of

[13:11](https://www.youtube.com/watch?v=SE6POOEjh4I&t=791s) Then uh we looked at the effect of expert guidance. So without expert

[13:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=793s) expert guidance. So without expert

[13:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=793s) expert guidance. So without expert guidance uh fewer tokens used fewer

[13:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=797s) guidance uh fewer tokens used fewer

[13:17](https://www.youtube.com/watch?v=SE6POOEjh4I&t=797s) guidance uh fewer tokens used fewer minutes spent on research but with

[13:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=800s) minutes spent on research but with

[13:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=800s) minutes spent on research but with expert guidance we see significantly

[13:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=802s) expert guidance we see significantly

[13:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=802s) expert guidance we see significantly more research being put into things. And

[13:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=805s) more research being put into things. And

[13:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=805s) more research being put into things. And in the case of Astra, we have a

[13:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=807s) in the case of Astra, we have a

[13:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=807s) in the case of Astra, we have a significant difference improvement in

[13:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=810s) significant difference improvement in

[13:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=810s) significant difference improvement in the predicted error.

[13:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=812s) the predicted error.

[13:32](https://www.youtube.com/watch?v=SE6POOEjh4I&t=812s) the predicted error. Finally, uh we wanted to see what

[13:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=815s) Finally, uh we wanted to see what

[13:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=815s) Finally, uh we wanted to see what happens if you don't give any consensus

[13:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=817s) happens if you don't give any consensus

[13:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=817s) happens if you don't give any consensus to these models. So we created a setting

[13:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=819s) to these models. So we created a setting

[13:39](https://www.youtube.com/watch?v=SE6POOEjh4I&t=819s) to these models. So we created a setting where we removed access to all the

[13:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=821s) where we removed access to all the

[13:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=821s) where we removed access to all the consensus. We even had like document

[13:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=824s) consensus. We even had like document

[13:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=824s) consensus. We even had like document extraction tools redact any mentions of

[13:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=827s) extraction tools redact any mentions of

[13:47](https://www.youtube.com/watch?v=SE6POOEjh4I&t=827s) extraction tools redact any mentions of consensus from any sort of news sources

[13:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=829s) consensus from any sort of news sources

[13:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=829s) consensus from any sort of news sources as well. And what we see here is that

[13:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=832s) as well. And what we see here is that

[13:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=832s) as well. And what we see here is that the the best models uh don't really need

[13:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=835s) the the best models uh don't really need

[13:55](https://www.youtube.com/watch?v=SE6POOEjh4I&t=835s) the the best models uh don't really need consensus anymore. So they are able to

[13:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=837s) consensus anymore. So they are able to

[13:57](https://www.youtube.com/watch?v=SE6POOEjh4I&t=837s) consensus anymore. So they are able to reconstruct the consensus view. Uh in

[13:59](https://www.youtube.com/watch?v=SE6POOEjh4I&t=839s) reconstruct the consensus view. Uh in

[14:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=840s) reconstruct the consensus view. Uh in fact in the case of Astra there is no

[14:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=842s) fact in the case of Astra there is no

[14:02](https://www.youtube.com/watch?v=SE6POOEjh4I&t=842s) fact in the case of Astra there is no significant difference between with and

[14:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=844s) significant difference between with and

[14:04](https://www.youtube.com/watch?v=SE6POOEjh4I&t=844s) significant difference between with and without consensus whereas for the older

[14:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=846s) without consensus whereas for the older

[14:06](https://www.youtube.com/watch?v=SE6POOEjh4I&t=846s) without consensus whereas for the older models you did like see a big jump with

[14:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=848s) models you did like see a big jump with

[14:08](https://www.youtube.com/watch?v=SE6POOEjh4I&t=848s) models you did like see a big jump with access to consensus. Um so yeah so we

[14:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=852s) access to consensus. Um so yeah so we

[14:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=852s) access to consensus. Um so yeah so we are seeing frontier models outperforming

[14:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=854s) are seeing frontier models outperforming

[14:14](https://www.youtube.com/watch?v=SE6POOEjh4I&t=854s) are seeing frontier models outperforming consensus. Uh so the question is you

[14:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=856s) consensus. Uh so the question is you

[14:16](https://www.youtube.com/watch?v=SE6POOEjh4I&t=856s) consensus. Uh so the question is you know what's next? Uh this is available

[14:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=858s) know what's next? Uh this is available

[14:18](https://www.youtube.com/watch?v=SE6POOEjh4I&t=858s) know what's next? Uh this is available for alpha testing in the product. But

[14:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=860s) for alpha testing in the product. But

[14:20](https://www.youtube.com/watch?v=SE6POOEjh4I&t=860s) for alpha testing in the product. But we're also working with customers to see

[14:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=862s) we're also working with customers to see

[14:22](https://www.youtube.com/watch?v=SE6POOEjh4I&t=862s) we're also working with customers to see if their internal data is useful for

[14:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=864s) if their internal data is useful for

[14:24](https://www.youtube.com/watch?v=SE6POOEjh4I&t=864s) if their internal data is useful for some of these predictive tasks. So feel

[14:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=865s) some of these predictive tasks. So feel

[14:25](https://www.youtube.com/watch?v=SE6POOEjh4I&t=865s) some of these predictive tasks. So feel free to get in touch. We are doing

[14:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=867s) free to get in touch. We are doing

[14:27](https://www.youtube.com/watch?v=SE6POOEjh4I&t=867s) free to get in touch. We are doing research on model reasoning. So I'm

[14:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=870s) research on model reasoning. So I'm

[14:30](https://www.youtube.com/watch?v=SE6POOEjh4I&t=870s) research on model reasoning. So I'm trying to understand if this reasoning

[14:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=871s) trying to understand if this reasoning

[14:31](https://www.youtube.com/watch?v=SE6POOEjh4I&t=871s) trying to understand if this reasoning is the same way as humans do their

[14:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=873s) is the same way as humans do their

[14:33](https://www.youtube.com/watch?v=SE6POOEjh4I&t=873s) is the same way as humans do their reasoning. If not, is there a way to

[14:35](https://www.youtube.com/watch?v=SE6POOEjh4I&t=875s) reasoning. If not, is there a way to

[14:36](https://www.youtube.com/watch?v=SE6POOEjh4I&t=876s) reasoning. If not, is there a way to complement human reasoning to try and

[14:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=877s) complement human reasoning to try and

[14:37](https://www.youtube.com/watch?v=SE6POOEjh4I&t=877s) complement human reasoning to try and you know get the best of both worlds. Um

[14:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=880s) you know get the best of both worlds. Um

[14:40](https://www.youtube.com/watch?v=SE6POOEjh4I&t=880s) you know get the best of both worlds. Um and then uh we're looking at

[14:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=881s) and then uh we're looking at

[14:41](https://www.youtube.com/watch?v=SE6POOEjh4I&t=881s) and then uh we're looking at post-raining and continual learning. So

[14:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=884s) post-raining and continual learning. So

[14:44](https://www.youtube.com/watch?v=SE6POOEjh4I&t=884s) post-raining and continual learning. So these agents just like humans right you

[14:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=886s) these agents just like humans right you

[14:46](https://www.youtube.com/watch?v=SE6POOEjh4I&t=886s) these agents just like humans right you can learn through experience. you can

[14:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=888s) can learn through experience. you can

[14:48](https://www.youtube.com/watch?v=SE6POOEjh4I&t=888s) can learn through experience. you can learn by the mistakes you've made in the

[14:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=889s) learn by the mistakes you've made in the

[14:49](https://www.youtube.com/watch?v=SE6POOEjh4I&t=889s) learn by the mistakes you've made in the past and so can the models keep learning

[14:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=892s) past and so can the models keep learning

[14:52](https://www.youtube.com/watch?v=SE6POOEjh4I&t=892s) past and so can the models keep learning from the mistakes in the past I think

[14:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=893s) from the mistakes in the past I think

[14:53](https://www.youtube.com/watch?v=SE6POOEjh4I&t=893s) from the mistakes in the past I think that's the other interesting question uh

[14:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=896s) that's the other interesting question uh

[14:56](https://www.youtube.com/watch?v=SE6POOEjh4I&t=896s) that's the other interesting question uh you have a couple of QR codes uh we have

[14:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=898s) you have a couple of QR codes uh we have

[14:58](https://www.youtube.com/watch?v=SE6POOEjh4I&t=898s) you have a couple of QR codes uh we have the blog post uh there's a lot more

[15:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=900s) the blog post uh there's a lot more

[15:00](https://www.youtube.com/watch?v=SE6POOEjh4I&t=900s) the blog post uh there's a lot more details there also a link to Frontier

[15:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=903s) details there also a link to Frontier

[15:03](https://www.youtube.com/watch?v=SE6POOEjh4I&t=903s) details there also a link to Frontier Finance it might be of interest if you

[15:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=905s) Finance it might be of interest if you

[15:05](https://www.youtube.com/watch?v=SE6POOEjh4I&t=905s) Finance it might be of interest if you are trying to evaluate your own sort of

[15:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=907s) are trying to evaluate your own sort of

[15:07](https://www.youtube.com/watch?v=SE6POOEjh4I&t=907s) are trying to evaluate your own sort of hardnesses um yeah and uh Arash and I

[15:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=912s) hardnesses um yeah and uh Arash and I

[15:12](https://www.youtube.com/watch?v=SE6POOEjh4I&t=912s) hardnesses um yeah and uh Arash and I will be here would love to chat with you

[15:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=913s) will be here would love to chat with you

[15:13](https://www.youtube.com/watch?v=SE6POOEjh4I&t=913s) will be here would love to chat with you thank