# Right Answer, Wrong Reasons: AI Forecasts Costco - Hassan El Mghari & Omar El Mghari, Together AI

Hassan El Mghari, Together AI · Omar El Mghari

[Talk page](https://ai.engineer/talks/dznCt8xPW7o-right-answer-wrong-reasons-ai-forecasts-costco) · [Video](https://www.youtube.com/watch?v=dznCt8xPW7o)

## Right Answer, Wrong Reasons: AI Forecasts Costco

Five forecasts of Costco’s profits had a reported average error of about 4.4%. In a separate audit test, replacing the guessed executive membership share with Costco’s disclosed figure pushed the error from about 1.7% to 28%.

### An eight-quarter forecast from before the fee announcement

[0:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=1s)

Could an open language model take Costco’s public filings and build a forecast of its profits over the next eight quarters? Hassan El Mghari, who led developer experience at Together AI, built the app with his father, Omar El Mghari, a former CFO and head of internal audit. When Hassan saw how closely the forecasts tracked Costco’s results, he thought the experiment had worked. Omar wanted to see the assumptions.

Membership pricing gave them a change worth modeling. Costco sold roughly $300 billion of merchandise a year at close to cost. Hassan put operating profit at about $12 billion and attributed a little over half of it to membership fees. In July 2024, Costco announced its first fee increase since 2017. Standard membership would go from $60 to $65 and executive membership from $120 to $130. Each executive renewal at the new price would therefore bring in twice as much additional cash as a standard renewal.

The team set the information cutoff a couple of weeks before the announcement. The question was conditional. Given what Costco had disclosed by then, how would revenue and profit change *if* the company raised its fees? The team could supply the pricing decision, calculate its consequences and compare the forecast with later results.

### Keeping hindsight out of the forecast

[4:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=252s)

Restricting the filings solved only part of the problem. Asked whether Costco had changed its membership fees in 2024, the first open models the team tried could identify the increase. Older candidates missed it. The team chose **GPT-OSS-120B** to build the financial models. The screening question could reveal prior knowledge, though a missed answer alone could establish little about everything a model had seen in training.

The working material came from about 214 public filings, including quarterly and annual earnings reports and sales reports. The team turned those documents into roughly 3,000 timestamped facts, then filtered on when each fact was available. Sixteen parallel sessions received the pre-cutoff material. Each session had to turn those facts into formulas the team could inspect and run.

### What a $65 payment contributes to a quarter

[5:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=344s)

Suppose a member pays $65 today. How much fee revenue belongs in this quarter? Costco’s quarterly report said the company recognizes membership fee revenue on a deferred basis. GPT-OSS-120B translated that policy into structured JSON specifying ratable recognition over a term of 52 weeks. Those two choices tell the calculation how to divide the payment and over what period.

At $65 divided by 52, each week of membership contributes $1.25 of revenue. The member pays once, while each quarter picks up only the portion earned during that quarter. Writing the rule as a formula let the language model interpret the filing once and left the arithmetic to the financial model.

Hassan laid the financial model out in Excel. It combined 14 generated formulas, figures taken directly from Costco’s filings and assumptions supplied by the language model. Two of those assumptions would matter especially. How many members were in the executive tier, where the increase was $10? How many were in the US and Canada, where the increase applied?

### Five of sixteen builds passed the engineering checks

[7:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=438s)

A formula that depended on itself failed a structural check. A build in which a higher fee failed to raise revenue failed a behavioral check for this scenario. When a check failed, the system sent the error back to the session that wrote the formulas and gave it up to two chances to repair the build.

Five builds passed and went into the forecast comparison. The other eleven still failed when their repair attempts ran out. The five thus qualified on the behavior of their formulas, before the team compared their forecasts with Costco’s reported results.

Hassan had started by tuning a single prompt, trying to get one build to pass all the checks. That took enough engineering time to make parallel attempts attractive. The sixteen runs together cost 19 cents. At that price he could afford to discard eleven candidates. What the other five had in common depended on what he had thought to test.

### Running the September scenario

[8:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=521s)

In the app, Hassan asked what would happen if Costco raised its fee that September. One of the newer language models had a narrower job here. It translated the question into structured scenario data and passed those inputs to the financial model GPT-OSS-120B had already written. Evaluating the formulas took about 10 milliseconds after translation. The app returned a projected **$560 million increase in profit over two years**.

How closely did predicted profit track the results Costco later reported? Hassan plotted the prediction in blue and the earnings-report figures in orange. He reported an average error of about 4.4% across the five passing builds, with individual errors ranging from 0.7% to 5.2%. The error formula was unspecified, which limits what that comparison tells us about the separate $560 million estimate of incremental profit. Hassan found the agreement encouraging enough to show his dad.

![Omar’s chart shows how much the output moves when a single assumption changes.](https://ai.engineer/api/talk-essay-media/84a2af78644d5857268ce5f58735be0ba5cdc2cbe23696d24b2b83c85f191fb0)

Omar varies one input at a time to see which assumptions most affect the forecast.

### A real citation for a guessed 15%

[9:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=594s)

Omar started by looking at which assumptions could move the forecast. Changing the executive share produced large swings. Changing the US-and-Canada assumption produced essentially no movement in sales or fees. That second result troubled him. If only members in those countries faced the increase, changing the number of members there should change the forecast.

Where did the executive share of 15% come from? Its citation led to a genuine statement that executive membership penetration had increased. That established a direction of change. The language model supplied the percentage itself, along with a range of 5% to 30%. Costco’s filings already disclosed 45%. An analyst could have tested every value in the proposed range and still missed the figure in the filing.

![The source quote about executive memberships is real but does not supply the percentage that the model inferred.](https://ai.engineer/api/talk-essay-media/1a6ab2f426313ffd36dbfefb2f82141287f4fcddf27f6331664d7f151bf8b946)

The filing discusses executive memberships. The quoted passage does not give the percentage used in the forecast.

### Fixing one assumption exposed the other error

[11:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=699s)

The audit traced the close result to two inputs that pushed the fee estimate in opposite directions.

- **Too many affected members.** The formulas applied the increase to Costco’s worldwide membership. Members outside the US and Canada contributed a price increase they would never pay. That explained why changing the geographic assumption had done so little.
- **Too few executive members.** At a 15% executive share, too few members paid the $10 increment and too many paid the $5 increment. The resulting average increase per member was too small, partly concealing the inflated member count.

Omar also compared estimates of the fee increase itself. Across the sixteen attempts, those estimates ranged from about $500 million to $6 billion. His recalculation put the increase at $450 million. Even the engineering check that required revenue to rise with price allowed plenty of room for a wrong answer.

What happens if you fix just the executive share? In Omar’s diagnostic example, the error started at about 1.7%. He replaced 15% with the disclosed 45% and left the worldwide member count in place. The error jumped to 28%. The formulas now assigned the larger increase to more members of a population that was already too large. Putting one assumption right made the remaining mistake easier to see.

![One panel identifies where prices were charged; the second compares executive share with the filed number. The third compares the result per member.](https://ai.engineer/api/talk-essay-media/7403f29a39817e9d67e0e22b686cd3560b43e604899c3d40b14626935ff6250a)

Changing the pricing assumption reveals that an error in the membership mix had pulled the estimate in the other direction.

### Turning accounting judgment into checks

[13:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=810s)

Omar borrowed three lines of defense from his auditing work to explain who could catch which failure.

- **Management controls.** Hassan’s engineering checks could catch a self-reference or a fee increase that moved revenue in the wrong direction.
- **Internal accounting review.** An auditor could trace a sensitive assumption back to the number in a filing, or change the affected member count and see whether the forecast responded. Both tests mattered here.
- **Independent review.** Omar proposed external auditors as a further check on the work of the people who built and reviewed the model.

The expert’s work could carry forward into the next build. Hassan and Omar turned several accounting checks into code, including one for the disclosed 45% executive share in this case. A future candidate could face that check automatically. Omar still saw a job for professional judgment. Someone familiar with the business had to notice that an input as important as the number of affected members could change without moving the forecast, then decide what the software should test.

## Key takeaways

- A historical forecast has two routes for hindsight. Filter company disclosures by when their facts became available, and check whether the language model already knows the event.
- Writing formulas makes a forecast inspectable. An auditor can examine the 52-week recognition rule, trace the tier mix to a disclosure and test whether changing the affected population moves revenue.
- When two errors may offset, correct one input and hold the other fixed. The jump to 28% error exposed how much the original result depended on understating the executive share while charging the increase worldwide.

## Slides

### Can AI reconstruct a company?

Hassan and Omar El Mghari gave open models everything Costco published before June 30, 2024. The models built a model of Costco’s business, and the presenters checked its profit forecasts against actual results for the next eight quarters.

[0:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=12s) [Hassan El Mghari and Omar El Mghari introduce an experiment using Costco publications from before June 30, 2024 to forecast the next two years.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c98deb777b3b63d0c828b50d244ae1d8b005b0c902573bf2d6837547d920adf.webp)

Can AI reconstruct a company?

We gave open models everything Costco published before June 30, 2024. They built a model of the business, and we checked its forecasts against the next two years.

Hassan El Mghari
TOGETHER AI

Omar El Mghari
CPA

together.ai
AI ENGINEER NYC · OCTOBER 13, 2026

### I build AI apps

Hassan El Mghari has built 40+ open-source AI apps with 3M+ users and 97K GitHub stars. TurboSeek.io lets people search, PDFtoChat lets them ask questions about papers, and CodeArena compares how well models code.

[0:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=48s) [Hassan El Mghari’s portfolio slide reports 40+ AI apps built, 3M+ users and 97K GitHub stars. Screenshots show apps for search, coding, image generation and document chat.](https://assets.ai.engineer/sites/aiecode2025/sha256/30f6cd9c2c1988000ea2dc23ca752ef01e5410691aaf40e12d12aec4cfa80ff0.webp)

I build AI apps
HASSAN EL MGHARI · DIRECTOR OF DEVELOPER EXPERIENCE, TOGETHER AI

40+ AI apps built | 3M+ users | 97K GitHub stars
github.com/nutlope

together.ai | WHO WE ARE
02 / 34

Interface text:
Turn ideas into comics
[illegible]

SubStudio
Generate Perfect Subtitles
[illegible]
Powered by together.ai

LlamaCoder
Powered by together.ai
Turn your idea into an app
Build me a budgeting app...
[illegible]

TurboSeek.io
The open source AI search engine.
Search smarter & faster.
Fully Open-Source

Loras
Generate AI images using different LoRAs/styles
Icons
Illustrations
Lee
Logos

CodeArena
OPEN SOURCE LLMS
Which LLM is best at coding?
Code Battle
[illegible]

SmartPDFs.io

[illegible]
BlinkShot
Built with together.ai
Enhance Prompt

LogoCreator
Generate your dream logo in a matter of seconds
[illegible]

PDFtoChat
Have a conversation with your papers, textbooks, and contracts for free!
Start Chatting Now!

Gloss
Highlight any line
Ask it anything.
Paste a link to any article
Read
On a Piece of Chalk
On Thinking for Oneself
Reading

### Together AI cloud platform

Hassan and Omar used Together AI to run the talk’s open source models. Developers can call models through its inference API, fine-tune them on their own data, or launch GPU clusters for training or inference.

[1:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=62s) [A numbered list describes running models through an inference API, fine-tuning models on your own data and launching on-demand GPU clusters.](https://assets.ai.engineer/sites/aiecode2025/sha256/0593a503270350d3fddae2e47d53b003ea2336a4bb2ccaa7bcd4b91ae47c0138.webp)

together.ai

We're an AI native cloud platform for running, deploying, and training open source models.
We make it easy to:

1. Run open source AI models (Inference API):
   • GLM-5.3 → chat & coding models
   • Nano Banana 2 → image models
   • Cartesia Sonic 3 → audio models
   • Kimi K3 → vision models
2. Finetune models on your own data (SFT, DPO, and RL)
3. Launch GPU clusters (H100s/B200s/B300s) on-demand to train your own models or run your own inference

TRUSTED BY INNOVATORS
CURSOR   Pika   Cartesia   LG AI Research   salesforce   wp

together.ai | WHO WE ARE
03 / 34

### I’ve spent 25 years checking other people’s numbers.

Omar El Mghari ran finance at a $1B pharmaceutical company, built an internal audit department and wrote reconciliation software. Now he brings that experience to checking his son Hassan’s work.

[1:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=71s) [Four columns summarize Omar El Mghari’s experience as a CFO, building an internal audit department, conducting SOX and SOC audits, and writing reconciliation software.](https://assets.ai.engineer/sites/aiecode2025/sha256/699e90d36ad5c27ad27ca789e9456e0f3bb16e2832bad668bc4c9f58795621fa.webp)

I've spent 25 years checking other people's numbers.

OMAR EL MGHARI · CPA, CIA, CISA, PMP, MBA

CFO
Ran finance at a $1B pharmaceutical company, with 32 accountants in five countries.

Audit
Built an internal audit department from scratch for a $500M construction company.

SOX · SOC
Audits on SAP, Oracle, NetSuite, Dynamics and QuickBooks.

Code
Six years of writing software, including pmsox.io, a reconciliation platform.

Today I'm checking Hassan's work.

together.ai | WHO WE ARE
04 / 34

### My first game was too slow. The rewrite was too fast.

In Omar’s BASIC tennis game, the ball took about a minute to cross the court. At 16, he rewrote the game in assembly and lost sight of the ball. He added a loop that counted down from 255 so he could see it move again.

[1:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=99s) [An Epson HX-20, an assembly-language book and two tennis-court diagrams illustrate a slow BASIC game. The assembly rewrite moved the ball too fast to see, so Omar added a delay loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9b6457db41accdaae30e8411fcf2983c3851ee88dfbe27c650e6a3779343cff.webp)

My first game was too slow.
The rewrite was too fast.

FUN FACT · RABAT, MOROCCO, 1984

MY BROTHER'S EPSON HX-20
EPSON HX-20
MICRO CASSETTE DRIVE

THE BOOK
INITIATION AU LANGAGE ASSEMBLEUR
[illegible]
50 PROGRAMMES PRATIQUES
8080   8085

BASIC
A tennis game: a rectangle court, two rackets of five dashes, and a line equation to bounce the ball. One crossing took about a minute.

Assembly, at 16
I learned registers and the accumulator from that book and rewrote the game. Now I couldn't see the ball.
?

The fix
       ld   b,255
wait:  djnz wait
Empty loops that do nothing, so a human could see the ball again.

My first speed fix made the program slower.

together.ai | WHO WE ARE
05 / 34

### Costco membership-fee forecast averaged 4.4% error

Using only information public in June 2024, the AI forecast Costco’s membership fees with a 4.4% average error over two years. When the presenter’s father checked the model, he found that its reasoning was wrong.

[2:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=139s) [A large 4.4% gives the average forecast error over two years using information public in June 2024. The presenter says their father checked the model and found it got the right answer for the wrong reasons.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ad3c5e2630b82cd5941fd7df4a9a154f487a38959c3491f8423fb896927c242.webp)

WHERE WE ENDED UP

4.4%

The AI's forecast of Costco's membership fees was off by 4.4% on average over two years. It only knew what was public in June 2024.

Then my dad checked the model. It got the right answer for the wrong reasons.

togetherai | OPENING
06 / 34

### Costco’s membership fees equal about half its operating profit

Costco sold $297B of merchandise at close to cost in fiscal 2026. Its $5.9B in membership fees equaled just over half of its $11.7B operating profit, which explains why the speakers focused their forecast on fees.

[3:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=182s) [Fiscal 2026 figures show $297 billion in net sales and $11.7 billion in operating profit. A purple bar spans about half the operating-profit total, representing $5.9 billion in membership fees.](https://assets.ai.engineer/sites/aiecode2025/sha256/5fb699d5abaac4dc94bae615b958af57ac778bc05e9415cafb15a8f5e282859d.webp)

WHY COSTCO

Costco sells $297B of stuff at close to cost.
Its membership fees equal about half its operating profit.

NET SALES · FISCAL 2026
$297B

OPERATING PROFIT
$11.7B

MEMBERSHIP FEES
$5.9B

OPERATING PROFIT, FISCAL 2026 · $11.7B
MEMBERSHIP FEES · $5.9B · ≈ 50%
EVERYTHING ELSE · $5.8B

together.ai | THE SETUP
07 / 34

### The timeline for forecasting Costco’s 2024 fee increase

Using only information public by June 30, 2024, ten days before Costco announced the fee increase, could an AI forecast how higher fees would affect revenue and profit? Eight quarters of results through May 2026 provided the comparison.

[3:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=212s) [The forecast uses information public by June 30, 2024. Costco announced increases from $60 to $65 and $120 to $130 on July 10. They took effect September 1. Eight reported quarters through May 2026 provide the comparison.](https://assets.ai.engineer/sites/aiecode2025/sha256/77134a544cbf80691535e5e7439bcee0e860eb13c64ebb0400f9f969a8dc0ba1.webp)

THE TEST
In July 2024, Costco raised its fee for the first time in seven years.

Timeline:
JUN 30, 2024 — CUTOFF — the AI sees nothing after
JUL 10, 2024 — ANNOUNCED — $60→$65 · $120→$130
SEP 1, 2024 — TAKES EFFECT — first since 2017
8 QUARTERS — REPORTED — the answer key, to May 2026

OUR QUESTION
Using only what was public on June 30, 2024, could an AI build a model of Costco and predict what the fee increase would do?

together.ai | THE SETUP
08 / 34

### Three model answers about Costco’s 2024 fee increase

With no documents, GLM-5.3 recalled that Costco’s fee rose from $60 to $65 on September 1, 2024. Llama cited its December 2023 cutoff. GPT-OSS got both fee changes right but put them on March 1.

[4:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=273s) [Asked when the increase took effect and by how much, GLM-5.3 recalled September 1 and the $60-to-$65 change. Llama 3.3 70B said its knowledge ended in December 2023. GPT-OSS-120B gave the correct amounts but incorrectly dated the increase to March 1.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4bfadad5e22febe36d073a307ac03c00d82a11d0c4bb888b4addf9e65c12f56.webp)

ONE OF THE QUESTIONS, ASKED WITH NO DOCUMENTS
“Did Costco change its membership fees in 2024? When did it take effect, and by how much?”

GLM-5.3 · RELEASED 2026
“On July 10, 2024, Costco announced its first membership fee increase since June 2017, effective September 1, 2024 ... $60 to $65.”
× REMEMBERS THE ANSWER

LLAMA 3.3 70B · TRAINED TO DEC 2023
“I do not have information about Costco’s membership fee changes in 2024. My knowledge cutoff is December 2023.”
✓ DOESN'T KNOW

GPT-OSS-120B · TRAINED TO JUN 2024
“Yes. Costco announced a membership-fee increase that took effect March 1, 2024 ... $60 to $65 ... $120 to $130.”
~ RIGHT AMOUNTS, WRONG DATE, LIKE 2017

togetherai | THE SETUP
10 / 34

### Lesson 1: Use a model trained before the forecast event

To test a historical forecast, choose a model whose training data ends before the event so it can’t recall the outcome. Give it only information public by the forecast date and turn off web search.

[4:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=291s) [Choose a model whose training data ends before the event being forecast. Give it only information that was public at the time and disable web search.](https://assets.ai.engineer/sites/aiecode2025/sha256/a77b4c7eb94c85b623f83ae8fd772c583cda272e9419477c534a0196f3c0d84c.webp)

LESSON 1

Use a model whose training data ends before the event you're forecasting.

Then give it only what was public at the time, with no web search.

togetherai
THE SETUP
11 / 34

### The model writes the formulas, and our code does all the math.

Using only facts public by June 30, 2024, 16 gpt-oss-120b sessions each wrote a Costco model. Code ran each model and sent errors back up to twice. The passing model with the lowest error over the year before the cutoff won.

[5:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=301s) [A flowchart traces 214 Costco filings to 3,310 timestamped facts. A June 30, 2024 cutoff limits the inputs to 16 model builds. Code checks each build, sends errors back up to twice and selects the passing build with the lowest error on the year before the cutoff.](https://assets.ai.engineer/sites/aiecode2025/sha256/b684e1f371d60722972a1988acd35b9a0a949561473d5962abf89c20709f8c6b.webp)

HOW IT WORKS
The model writes the formulas, and our code does all the math.

COLLECT THE FILINGS · PLAIN CODE
LLM    PLAIN CODE

214 filings
10-K · 10-Q · 8-K · monthly sales

3,310 facts
stamped with SEC publish time

Cut at June 30, 2024
drop anything published later

What the model sees

BUILD · THE MODEL WRITES, OUR CODE CHECKS

16 × gpt-oss-120b
trained only up to June 2024
each writes a model of Costco

Checks
run each model
reject it with a reason

ERRORS GO BACK TO THE MODEL · UP TO 2 TIMES

Pick a winner
lowest error on the year before the cutoff, rule set in advance

togetherai    BUILDING THE MODEL
12 / 34

### How the model writes a formula for membership revenue

The model turns a sentence from Costco’s quarterly report into a 52-week formula and cites the sentence it used. The code runs the formula: one $65 payment in September yields about $5.42 in revenue each month through August.

[5:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=341s) [Three panels connect Costco’s quarterly report to a JSON formula and its result. The formula spreads an annual fee over 52 weeks. A chart shows one $65 September payment and approximately $5.42 in recognized revenue each month through August.](https://assets.ai.engineer/sites/aiecode2025/sha256/6cae3f54f5aeff50b47827c7c467666dbec2a93c39d8b83cc44cecfa39aa63bd.webp)

WHAT THE MODEL WRITES
The model writes formulas, like a spreadsheet.
Here's the one for membership revenue.

1 · COSTCO'S QUARTERLY REPORT SAYS
“We account for membership fee revenue on a deferred basis, recognized ratably over the one-year membership period.”

In plain English: members pay for a year up front, and Costco counts it as revenue a little each month.

2 · THE MODEL WRITES THIS JSON (OUR NOTES IN GRAY)
{ "id": "recognized_membership_fees",
  "formula": {
    // spread evenly over 52 weeks…
    "op": "ratable", "term_weeks": 52,
    // …the cash that members paid
    "arg": { "ref": "cash_membership_fee" } },
  // and here's the sentence that says so
  "evidence": "recognized ratably over the one-year membership period" }

3 · OUR CODE RUNS IT
CASH IN      $65 once
REVENUE      $5.42 a month
Timeline: SEP    AUG

$65 paid in September becomes $5.42 of revenue every month for a year.

The model never does the arithmetic. If you ask a chatbot instead, you get a paragraph you can't run or check.

togetherai | BUILDING THE MODEL
13 / 34

### What the model had to assume

The model filled gaps in Costco’s data with estimates. It put 15% of members on the $120 Executive tier (range 5–30%), placed 60% of members in the US and Canada (50–70%), and attributed 10% of net sales to gasoline (5–20%).

[6:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=393s) [Three cards show the model’s estimates. It assumes 15% of members are on the Executive tier, with a range of 5–30%. It places 60% of members in the US and Canada, with a range of 50–70%, and attributes 10% of sales to gasoline, with a range of 5–20%.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f762be16e1b143b2a170fa6cbe3db00b372ad2aab718a9466ef58b3696460cb.webp)

WHAT THE MODEL HAD TO GUESS
Some numbers weren't in the data we gave it,
so the model assumed them.

MEMBERS ON THE $120 EXECUTIVE TIER
15%
Scale: 0%, 100%
its range: 5–30%
ITS REASON
“Typical share of Executive members (assumed).”

MEMBERS IN THE US AND CANADA
60%
Scale: 0%, 100%
its range: 50–70%
ITS REASON
“Portion of members residing in U.S. and Canada (assumed).”

GAS AS A SHARE OF SALES
10%
Scale: 0%, 100%
its range: 5–20%
ITS REASON
“Estimated share of net sales attributable to gasoline (assumed).”

Three of its eight assumptions. It also listed six things it said it couldn't know from the filings.

togetherai | BUILDING THE MODEL
14 / 34

### The full Costco model

Read the Excel model left to right. Assumptions about member mix and fee increases feed into membership revenue. The graph connects revenue and costs to operating income and distinguishes 14 formulas from seven figures in Costco’s filings.

[6:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=412s) [A left-to-right graph connects scenario assumptions to membership, revenue, costs and operating income. The legend identifies eight assumptions in orange, 14 formulas in green and seven figures from Costco’s filings in blue.](https://assets.ai.engineer/sites/aiecode2025/sha256/d18ffa4c268f29dcf3c63381162f5d3a54ace01cd27f46aff9170faeb2566446.webp)

THE WHOLE MODEL
Here's the full picture.

SCENARIO LEVERS 8
Executive membership share — Assumed 15.0%
US/CA member share — Assumed 60.0%
Standard fee increase (USD) — Assumed $0
Executive fee increase (USD) — Assumed $0
Fee increase effective date — Assumed 2024-07-01
Renewal response to fee increase — Assumed 0.0 pts
Gas price change lever — Assumed 0.0%
Gasoline sales share of net sales — Assumed 10.0%

DRIVERS 1
Gas price impact on net sales (ratio) — Derived 0.0%

MEMBERSHIP 9
Paid members (thousands) — Reported 74.5M
Paid members YoY growth — Derived 7.8%
Paid members growth trend — Derived 7.7%
Standard annual fee (USD) — Derived $60
Executive annual fee (USD) — Derived $120
US/CA paid members (units) — Derived 44,700,000
Cash collected for membership fees (USD) — Derived $1.19B
Recognized membership fee... — Derived $1.15B
Membership fee revenue — Reported $1.12B

REVENUE 4
Net sales YoY growth — Derived 9.1%
Net sales growth trend — Derived 5.8%
Net sales — Reported $57.39B
Total revenue — Reported $58.52B

COSTS 6
Merchandise cost ratio — Derived 89.2%
Merchandise cost ratio trend — Derived 89.3%
Merchandise costs — Reported $51.17B
SG&A ratio — Derived 9.0%
SG&A ratio trend — Derived 9.2%
SG&A expenses — Reported $5.14B

PROFIT 1
Operating income — Reported $2.20B

8 ASSUMPTIONS, WITH ITS OWN RANGES
14 FORMULAS
7 NUMBERS FROM COSTCO'S FILINGS

togetherai | BUILDING THE MODEL
15 / 34

### Our code checks every model by running it.

Raising Costco’s Executive fee by $10 should bring in more membership revenue. The code caught a forecast where it didn’t. A check against the last four reported quarters found a 13.1% average error in membership fees, above the 5% limit.

[7:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=436s) [Four panels show failures caught by the checks. A formula depends on itself, a build forecasts a $163 billion quarterly loss, a $10 Executive fee increase leaves forecast membership fees unchanged, and a backtest misses four quarters by 13.1% on average against a 5% limit.](https://assets.ai.engineer/sites/aiecode2025/sha256/a5752c890c8ef6ce57bcb36ee062e7dcaf6edf2cefb04bc06d1b1adffeb097dc.webp)

CHECKING EVERY MODEL
Our code checks every model by running it.

A formula can't depend on itself.
A REAL ERROR FROM OUR RUN
base_fee_per_member → membership_fees → cash_membership_fee → base_fee_per_member

The forecast has to be believable.
A REAL ERROR FROM OUR RUN
operating_income forecast for FY2024Q4 is -163,554,529,799, -5981% vs the same quarter a year earlier
That build had Costco losing $163 billion in one quarter.

Raising a fee has to raise revenue.
A REAL ERROR FROM OUR RUN
a +$10 Executive fee alone does not raise forecast membership fees; wire fee_increase_executive_usd into membership cash for the members who pay it

It has to get last year right.
A REAL ERROR FROM OUR RUN
membership_fees: holding back the last 4 reported quarters, the formulas miss them by 13.1% on average (limit 5%)

Other checks: every citation has to exist in the data and be published before June 30, 2024, and a September increase can't change earlier quarters.

together.ai | BUILDING THE MODEL
16 / 34

### 16 builds at once — run in progress

Six seconds into the replay, all 16 builds are using the same Costco filings to build independent financial models. Build #04 has started writing formulas while the other visible builds are still reasoning.

[7:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=470s) [At six seconds, all 16 builds are running. Most visible tiles are labeled Reasoning, and one is labeled Writing.](https://assets.ai.engineer/sites/aiecode2025/sha256/af6b1bf6f32b2c10fc7209d6d6aa2249b8380536387cf7d23aaf4a6694bfb1a6.webp)

16 builds at once
SAVED RUN, SPED UP · SPACE PAUSES
WATCH FOR  green tiles passed · red tiles say why they failed · the star on the winner

16 independent builds, one packet
Each build samples its own model. Validators keep only models that reconcile, wire their levers, and reproduce the year before the cutoff. The lowest pre-cutoff backtest error wins.

ELAPSED 6s
PASSED 0
REJECTED 0
RUNNING 16
NODES WRITTEN 1
COST $0.00

#01 · Reasoning · 0 nodes · 9.2k chars reasoning
#02 · Reasoning · 0 nodes · 8.9k chars reasoning
#03 · Reasoning · 0 nodes · 9.5k chars reasoning
#04 · Writing · 1 nodes · 6.7k chars reasoning
#05 · Reasoning · 0 nodes · 7.4k chars reasoning
#06 · Reasoning · 0 nodes · 9.1k chars reasoning
#07 · Reasoning · 0 nodes · 8.9k chars reasoning
#08 · Reasoning · 0 nodes · 9.1k chars reasoning
#09 · Reasoning · 0 nodes · 9.4k chars reasoning
#10 · Reasoning · 0 nodes · 8.8k chars reasoning
#11 · Reasoning · 0 nodes · 8.6k chars reasoning
#12 · Reasoning · 0 nodes · 7.8k chars reasoning

WHAT THE BUILDER READS
178 sourced facts · 9 filing passages · history FY2021Q4–FY2024Q3
0 FACTS CITED

MEMBERSHIP RECOGNITION  Form 10-Q — quarterly repo...
We account for membership fee revenue on a deferred basis, recognized ratably over the one-year membership period.

RENEWAL RATE  Form 10-Q — quarterly report for period ...
At the end of the third quarter of 2024, our renewal rates were 93.0% in the U.S. and Canada and 90.5% worldwide.

COMPARABLE SALES DEFINITION  Form 10-Q — quarterl...
Comparable sales is 18 Table of Contents defined as net sales from warehouses open for more than one year, including remodels, relocations and expansions, and sales related to e-commerce sites operating for more than one year.

GASOLINE  Form 10-Q — quarterly report for period ende...
Generally, rising gasoline prices benefit net sales growth which, given the higher sales base, negatively impacts our gross margin percentage but decreases our SG&A expenses as a percentage

together.ai | BUILDING THE MODEL
18 / 34

Interface text:
CompanyLab / Costco COST
CAN AI RECONSTRUCT A COMPANY?
BUILD · SCENARIOS · TIME TRAVEL · MARKET
INFO CUTOFF  Jun 30, 2024
MODEL  openai/gpt-oss-120b
SAVED RUN · OFFLINE
INFORMATION CUTOFF  06/30/2024
BUILDER MODEL  Pinned · gpt-oss-120b (cutoff Jun 2024)
STRATEGY  16 PARALLEL BUILDS · SINGLE BUILD
☑ Replay saved run
BUILDING…  STOP
SAVED RUN

### 16 builds at once — results

Five of 16 Costco models passed the checks. The red tiles explain why the other 11 failed. Of the five green tiles, build #05 earned the star for the lowest error on the year before the cutoff. All 16 builds cost $0.19.

[8:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=485s) [A grid of 16 builds highlights build 05 as selected. The summary reports five passes, 11 rejections and a total cost of $0.19.](https://assets.ai.engineer/sites/aiecode2025/sha256/1084934bfc3d38590e1751ce84e670a426d3fbf83e83e3f404cbcd719ec5b139.webp)

16 builds at once
SAVED RUN, SPED UP · SPACE PAUSES
WATCH FOR  green tiles passed · red tiles say why they failed · the star on the winner

16 independent builds, one packet
Each build samples its own model. Validators keep only models that reconcile, wire their levers, and reproduce the year before the cutoff. The lowest pre-cutoff backtest error wins.

RECORDED 360s
PASSED 5
REJECTED 11
RUNNING 0
NODES WRITTEN 402
COST $0.19

#01 × REJECTED · Missing ratable (deferred) fee rec...
#02 ✓ PASSED · Backtest · sales 1.2% · fees 2.5%
#03 × REJECTED · Fee lever moves too much
#04 ✓ PASSED · Backtest · sales 3.9% · fees 1.3%
#05 ★ SELECTED · Backtest · sales 1.2% · fees 1.7%
#06 × REJECTED · lever timing
#07 × REJECTED · Missing ratable (deferred) fee rec...
#08 × REJECTED · Missing ratable (deferred) fee rec...
#09 × REJECTED · Misses pre-cutoff membership fe...
#10 × REJECTED · Misses pre-cutoff membership fe...
#11 ✓ PASSED · Backtest · sales 1.2% · fees 4.2%
#12 ✓ PASSED · Backtest · sales 1.2% · fees 3.7%
#13 × REJECTED · Missing ratable (deferred) fee rec...
#14 × REJECTED · Stopped at the deadline
#15 × REJECTED · Stopped at the deadline
#16 × REJECTED · No model returned

WHAT THE BUILDER READS
178 sourced facts · 9 filing passages · history FY2021Q4–FY2024Q3
60 FACTS CITED

GASOLINE  Form 10-Q — quarterly report for period ende...
Generally, rising gasoline prices benefit net sales growth which, given the higher sales base, negatively impacts our gross margin percentage but decreases our SG&A expenses as a percentage of net sales.
→ Gas price change lever 13/16
→ Gas effect multiplier 3/16
→ Net sales 2/16
→ Gas sales share of net sales 2/16
+4 more

EXECUTIVE MEMBERSHIP  Form 10-Q — quarterly report f...
The extent to which we achieve growth in our membership base, increase the penetration of Executive memberships, and sustain high renewal rates materially influences our profitability.
→ Executive fee increase (USD) 7/16
→ Renewal response (points) 5/16
→ Average executive fee (USD per member per year) 1/16
→ Cash collected from executive members 1/16
+7 more

together.ai | BUILDING THE MODEL
18 / 34

Interface text:
CompanyLab  CAN AI RECONSTRUCT A COMPANY? / Costco  COST
BUILD · SCENARIOS · TIME TRAVEL · MARKET
INFO CUTOFF  Jun 30, 2024
MODEL  openai/gpt-oss-120b
SAVED RUN · OFFLINE
SAVED RUN

### Instead of tuning one prompt, run many and let your code pick.

Across five runs, a carefully tuned prompt never produced a financial model that passed every check. The team then generated 16 candidates for $0.19 total. Five passed the automated checks, the first in about three minutes.

[8:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=492s) [One carefully tuned prompt passed all checks in zero of five runs. Five of 16 parallel builds passed, the first in about three minutes. All 16 cost $0.19.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4a9ac7d4aba4d75a356804d9f60df339102ffbe9f7a76e74f62c6552ca7817f.webp)

LESSON 2
Instead of tuning one prompt, run many and let your code pick.

ONE CAREFUL PROMPT
Never passed
we ran it 5 times · 0 passed every check

16 CHEAP TRIES, THEN OUR CHECKS
5 of 16 passed
$0.19 for all sixteen · first one passed in about 3 minutes

Have the model write something your code can run, run it many times, and let your checks throw out the bad ones.

together.ai
BUILDING THE MODEL
19 / 34

### A second model turns your question into settings.

GLM-5.3 translates the Costco question into fee increases of $5 for standard members and $10 for Executive members, effective September 1, 2024. The code uses fixed formulas to project $560 million in added membership fees over two years.

[8:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=519s) [With formulas fixed, the model sets Costco’s standard fee increase to $5 and its Executive fee increase to $10, effective September 1, 2024. The code projects $560 million in additional membership fees over two years.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea000a0e624f0f406478086a5e38faf4494233c9dbba412b624fd1adf323e871.webp)

ASKING A WHAT-IF
A second model turns your question into settings.
It can't touch the formulas.

YOU ASK
“What if Costco raises the fee $5, and $10 for Executive members, this September?”

GLM-5.3 →

IT CHANGES THREE SETTINGS
SETTING | BEFORE | AFTER
Standard fee increase | $0 | $5
Executive fee increase | $0 | $10
Starts on | | Sep 1, 2024

These are the orange boxes from the model. If a question needs a setting the model doesn't have, it says so.

OUR CODE
10 MS →

YOU GET
+$560M
in membership fees over two years

together.ai | TESTING IT
20 / 34

### The what-if, in the app

GLM-5.3 turns the Costco question into a proposal: raise the standard fee by $5 and the Executive fee by $10 on September 1, 2024. Before the user applies it, the chart shows only the baseline, which reaches $111M per week by FY2026 Q3.

[9:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=543s) [The scenarios dashboard shows a proposed $5 standard fee increase and $10 Executive fee increase. The chart shows the baseline, and additional membership fees still read $0 before the changes are applied.](https://assets.ai.engineer/sites/aiecode2025/sha256/542d5ad585a792d0fb37a8243412b678f904d7b8d4c00fee968ab5c22456de0f.webp)

The what-if, in the app
WATCH FOR  the settings it picks · the fee line climbing slowly instead of jumping

[02] SCENARIOS
Ask what-if, get a checkable patch
The model turns a question into explicit changes to named levers. You review the patch, then the typed graph recomputes in milliseconds.

Ask a what-if
Nothing changes until you apply the patch.
What if Costco raises the fee $5, and $10 for Executive members, this September?
INTERPRET
renewal rates drop 1 point
gas prices rise 30%

OFFLINE PRESETS · AUTHORED ASSUMPTIONS
FEE +$5 / +$10
RENEWAL -1 PT
GAS +30%

PROPOSED PATCH · ZAI-ORG/GLM-5.3
Costco fee increase effective September 2024
Standard fee increase (USD) = $5
all forecast quarters
Executive fee increase (USD) = $10
all forecast quarters

EXTRA MEMBERSHIP FEES · HORIZON
$0
cumulative through FY2026Q3

EXTRA OPERATING INCOME · HORIZON
$0
recomputed in 1430 ms

BREAK-EVEN RENEWAL DROP
—
Apply a fee change to compute

Membership fees per week, by quarter
Baseline. Apply a scenario to compare.
← REPORTED   FORECAST →
Y-axis ticks: $110M/wk, $100M/wk, $90.0M/wk
Endpoint: $111M/wk

Interface text:
SAVED RUN · SPACE PAUSES
CompanyLab / Costco  COST
CAN AI RECONSTRUCT A COMPANY?
BUILD  SCENARIOS  TIME TRAVEL  MARKET
INFO CUTOFF  Jun 30, 2024
MODEL  openai/gpt-oss-120b
SAVED RUN
MEMBERSHIP FEES  TOTAL REVENUE  OPERATING INCOME
PER WEEK  QUARTER TOTAL  TABLE
together.ai | TESTING IT
21 / 34

[9:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=551s) [The proposed changes are a $5 standard fee increase and a $10 Executive fee increase effective September 1, 2024. The baseline chart ends at $111 million per week, and the added-fee total still reads $0.](https://assets.ai.engineer/sites/aiecode2025/sha256/61e818b7b8c5272d63b5e90ebf02242300509f8a9d41aa6b2e152b3c9dc50282.webp)

The what-if, in the app
WATCH FOR  the settings it picks · the fee line climbing slowly instead of jumping

Ask a what-if
Nothing changes until you apply the patch.
What if Costco raises the fee $5, and $10 for Executive members, this September?
INTERPRET
renewal rates drop 1 point
gas prices rise 30%

OFFLINE PRESETS · AUTHORED ASSUMPTIONS
FEE +$5 / +$10
RENEWAL -1 PT
GAS +30%

PROPOSED PATCH · ZAI-ORG/GLM-5.3
Costco fee increase effective September 2024
Standard fee increase (USD) = $5
all forecast quarters
Executive fee increase (USD) = $10
all forecast quarters
Fee increase effective date = 2024-09-01
all forecast quarters
• Fee increase takes effect September 1, 2024 (the question says ‘this September’; model date is 2024-06-30).
• Renewal response to the fee increase is left unchanged (renewal_response_pts stays at 0); the question does not mention any member reaction.
• Executive membership share and US/CA member

$0
cumulative through FY2026Q3
$0
recomputed in 1430 ms
—
Apply a fee change to compute

Membership fees per week, by quarter
Baseline. Apply a scenario to compare.
← REPORTED   FORECAST →
Y-axis ticks: $110M/wk, $100M/wk, $90.0M/wk, $80.0M/wk
Timeline: ’23Q4, ’24Q2, ’24Q4, ’25Q2, ’25Q4, ’26Q2, ’26Q3
Endpoint: $111M/wk
Legend: Baseline

Assumptions
Levers the model exposes. Ranges are assumptions, not probabilities.
Executive membership share  15.0%
US/CA member share  60.0%

Interface text:
SAVED RUN · SPACE PAUSES
CompanyLab / Costco  COST
CAN AI RECONSTRUCT A COMPANY?
BUILD  SCENARIOS  TIME TRAVEL  MARKET
INFO CUTOFF  Jun 30, 2024
MODEL  openai/gpt-oss-120b
SAVED RUN
MEMBERSHIP FEES  TOTAL REVENUE  OPERATING INCOME
PER WEEK  QUARTER TOTAL  TABLE
RESET TO BASELINE
together.ai | TESTING IT
21 / 34

### Projected impact of the Costco fee increase

The model recognizes Costco’s higher fees over 12 months, so weekly fee revenue climbs gradually to $120M versus $111M at baseline. Through FY2026 Q3, it projects $560M in extra membership fees and the same gain in operating income.

[9:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=556s) [The app projects $560 million in additional membership fees and $560 million in additional operating income through FY2026 Q3. Weekly membership fees reach $120 million in the scenario, compared with $111 million at baseline.](https://assets.ai.engineer/sites/aiecode2025/sha256/6adac13623c9cc6ab678e060282d764d6e7082a4aec8b7272ebfd289ebff114c.webp)

The what-if, in the app
WATCH FOR  the settings it picks · the fee line climbing slowly instead of jumping

[02] SCENARIOS
Ask what-if, get a checkable patch
The model turns a question into explicit changes to named levers. You review the patch, then the typed graph recomputes in milliseconds.

Ask a what-if
Nothing changes until you apply the patch.
What if Costco raises the fee $5, and $10 for Executive members, this September?
INTERPRET
renewal rates drop 1 point
gas prices rise 30%

OFFLINE PRESETS · AUTHORED ASSUMPTIONS
FEE +$5 / +$10
RENEWAL -1 PT
GAS +30%

PROPOSED PATCH · ZAI-ORG/GLM-5.3
Costco fee increase effective September 2024
Standard fee increase (USD) = $5
all forecast quarters
Executive fee increase (USD) = $10
all forecast quarters

EXTRA MEMBERSHIP FEES · HORIZON
$560M
cumulative through FY2026Q3

EXTRA OPERATING INCOME · HORIZON
$560M
recomputed in 25 ms

BREAK-EVEN RENEWAL DROP
7.7 pts
Costco could lose 7.7 pts of renewal before added membership income reaches zero

Membership fees per week, by quarter
Baseline vs scenario. Fees are recognized over 12 months, so a change ramps instead of jumping.
← REPORTED   FORECAST →
Y-axis ticks: $120M/wk, $110M/wk, $100M/wk
Line endpoints: $120M/wk, $111M/wk

Interface text:
SAVED RUN · SPACE PAUSES
CompanyLab
CAN AI RECONSTRUCT A COMPANY? / Costco  COST
BUILD  SCENARIOS  TIME TRAVEL  MARKET
INFO CUTOFF  Jun 30, 2024
MODEL  openai/gpt-oss-120b
SAVED RUN
MEMBERSHIP FEES  TOTAL REVENUE  OPERATING INCOME
PER WEEK  QUARTER TOTAL  TABLE
together.ai | TESTING IT
21 / 34

### Checking the frozen forecast against eight reported quarters

Costco’s reported weekly figures rose in all eight quarters but stayed below the June 2024 forecast. The selected build missed by 4.4% on average. Six of the eight quarters fell within the range of the five builds that passed validation.

[9:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=560s) [Eight orange dots show Costco’s reported weekly figures below the blue forecast line. Six dots fall within the shaded range of the five validated builds. The selected build’s error over the eight quarters is 4.4%.](https://assets.ai.engineer/sites/aiecode2025/sha256/23d8bbf04861617a1a68fd69407142ca46e2ad37c98f809ff5343ee6a5e92fd6.webp)

Time travel: checking the forecast
FROM THE APP · FROZEN JUNE 2024 FORECAST VS WHAT COSTCO REPORTED
WATCH FOR  blue line the frozen forecast · orange dots Costco's real quarters, one at a time · band the five builds that passed

[03.1] THE REVEAL · SUMMARY
Eight quarters later
Built Jun 30, 2024 by openai/gpt-oss-120b, before the increase was announced, then given the real terms: +$5 standard, +$10 Executive from Sept 1, 2024. Dots are what Costco later reported, per week.

← REPORTED   FORECAST →
Y-axis ticks: $120M/wk, $110M/wk, $100M/wk, $90.0M/wk, $80.0M/wk
Timeline: ’23Q4, ’24Q1, ’24Q2, ’24Q3, ’24Q4, ’25Q1, ’25Q2, ’25Q3, ’25Q4, ’26Q1, ’26Q2, ’26Q3
Final reported point: $114M/wk
Legend:
Selected build, with increase · frozen Jun 30, 2024
Reported
Range of all 5 builds that passed validation

SELECTED BUILD · ERROR SO FAR
4.4%
over 8 reported quarters

VALIDATED BUILDS · RANGE
6 / 8
reported quarters inside the range of all 5 builds that passed

Interface text:
together.ai | TESTING IT
22 / 34

### The selected build’s average miss was 4.4%

Over eight quarters, the chosen forecast missed Costco’s reported profit by 4.4% on average, compared with 9.5% for an earlier build. The best of the five passing builds missed by 0.7%. Their forecasts bracketed 6 of 8 reported results.

[9:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=574s) [Four cards compare the selected build’s 4.4% average miss over eight quarters, an earlier build’s 9.5% miss, the 0.7% to 5.2% range across five passing builds, and six of eight reported quarters inside the band.](https://assets.ai.engineer/sites/aiecode2025/sha256/c66134c3c409b15418a869c442b6c711ec17cacd9d50792567fc3c988724e392.webp)

THE RESULT
Eight quarters later, it was off by 4.4%.

THE BUILD WE PICKED
4.4%
average miss, 8 quarters

AN EARLIER BUILD
9.5%
before we added stricter checks

THE 5 BUILDS THAT PASSED
0.7–5.2%
the best one wasn't our pick

INSIDE THE BAND
6 of 8
reported quarters

together.ai
TESTING IT
23 / 34

### Can we trust it?

Hassan felt good about the Costco forecast after the chosen build passed every check his team wrote. Then he showed it to his dad, who started with the assumptions behind the forecast.

[9:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=581s) [The presenter says the selected build passed every check they wrote. A separate line in orange reads, “Then I showed it to my dad.”](https://assets.ai.engineer/sites/aiecode2025/sha256/f63d32ec4a7976173f248f6054d49d9bb4682fd51d9fb79f65b67d250d999255.webp)

CAN WE TRUST IT?

The build we picked passed every check we wrote, and I was pretty happy with it.

Then I showed it to my dad.

together.ai | TESTING IT
24 / 34

### Which assumptions move the answer the most?

At 5% Executive share, the model cuts eight quarters of operating profit by $0.94B. At 30%, it adds $1.41B. Varying US/Canada member share from 50% to 70% changes profit by $0, even though the fee increase applies there.

[9:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=593s) [Horizontal bars show changes in eight quarters of operating profit when one assumption varies and the others stay fixed. Executive membership share shifts profit by −$0.94 billion to +$1.41 billion. Varying US/Canada member share from 50% to 70% has zero modeled effect.](https://assets.ai.engineer/sites/aiecode2025/sha256/84a2af78644d5857268ce5f58735be0ba5cdc2cbe23696d24b2b83c85f191fb0.webp)

Which assumptions move the answer the most?
OMAR EL MGHARI · CPA · WHERE I START

Standard fee increase
$0 – $20   +$1.88B

Executive share
5% – 30% · AI's range   −$0.94B   +$1.41B

Gas price change
−50% – +50%   −$1.26B   +$1.36B

Executive fee increase
$0 – $40   +$0.66B

Renewal response
−3 pts – 0   −$0.23B

US/Canada member share
50% – 70%   $0 · moves nothing

Legend: LOW END OF RANGE   HIGH END OF RANGE

HOW TO READ IT
Move one assumption from the low end of its range to the high end, keep the rest fixed, and see how much eight quarters of operating profit changes.

TWO THINGS TO LOOK AT
The Executive share is one of the widest bars. The US and Canada share, where the increase applies, moves nothing.

together.ai   THE AUDIT
25 / 34

## Full transcript

[0:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=12s) All right. Hello everybody. Thank you so

[0:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=14s) All right. Hello everybody. Thank you so

[0:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=14s) All right. Hello everybody. Thank you so much for coming. My name is Hassan. Uh

[0:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=16s) much for coming. My name is Hassan. Uh

[0:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=16s) much for coming. My name is Hassan. Uh this is my dad Omar. And uh we're very

[0:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=18s) this is my dad Omar. And uh we're very

[0:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=18s) this is my dad Omar. And uh we're very excited to be here today uh to to talk.

[0:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=21s) excited to be here today uh to to talk.

[0:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=21s) excited to be here today uh to to talk. Uh I think the only father-son talk at

[0:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=23s) Uh I think the only father-son talk at

[0:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=23s) Uh I think the only father-son talk at this conference. So um thank you.

[0:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=26s) this conference. So um thank you.

[0:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=26s) this conference. So um thank you. [cheering] Thank you. [applause]

[0:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=26s) [cheering] Thank you. [applause]

[0:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=26s) [cheering] Thank you. [applause] Thanks.

[0:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=29s) So, I'm here to talk about, you know,

[0:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=31s) So, I'm here to talk about, you know,

[0:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=31s) So, I'm here to talk about, you know, can LLMs actually uh make financial

[0:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=35s) can LLMs actually uh make financial

[0:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=35s) can LLMs actually uh make financial forecasts of a company? And so, we used

[0:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=36s) forecasts of a company? And so, we used

[0:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=36s) forecasts of a company? And so, we used Costco. We gave an open model kind of

[0:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=39s) Costco. We gave an open model kind of

[0:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=39s) Costco. We gave an open model kind of everything uh that was public from

[0:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=41s) everything uh that was public from

[0:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=41s) everything uh that was public from Costco's filings up until a certain

[0:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=43s) Costco's filings up until a certain

[0:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=43s) Costco's filings up until a certain point, and we had it try to predict

[0:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=44s) point, and we had it try to predict

[0:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=44s) point, and we had it try to predict Costco's profits over the next eight

[0:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=47s) Costco's profits over the next eight

[0:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=47s) Costco's profits over the next eight quarters. So, we'll start off with some

[0:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=49s) quarters. So, we'll start off with some

[0:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=49s) quarters. So, we'll start off with some intros. My name is Hassan. Uh I lead

[0:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=51s) intros. My name is Hassan. Uh I lead

[0:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=51s) intros. My name is Hassan. Uh I lead developer experience at Together AI. The

[0:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=53s) developer experience at Together AI. The

[0:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=53s) developer experience at Together AI. The main thing to know about me is I really

[0:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=54s) main thing to know about me is I really

[0:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=54s) main thing to know about me is I really like building AI apps. Everything I do

[0:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=56s) like building AI apps. Everything I do

[0:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=56s) like building AI apps. Everything I do is open source on GitHub. And uh this is

[0:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=58s) is open source on GitHub. And uh this is

[0:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=58s) is open source on GitHub. And uh this is another one of the apps that I built uh

[0:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=59s) another one of the apps that I built uh

[1:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=60s) another one of the apps that I built uh but this one was in collaboration uh

[1:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=61s) but this one was in collaboration uh

[1:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=61s) but this one was in collaboration uh with my dad and I work over at Together

[1:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=63s) with my dad and I work over at Together

[1:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=64s) with my dad and I work over at Together AAI. We used it for the open source

[1:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=65s) AAI. We used it for the open source

[1:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=65s) AAI. We used it for the open source models in in this talk. We're a platform

[1:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=67s) models in in this talk. We're a platform

[1:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=67s) models in in this talk. We're a platform for uh running and deploying and

[1:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=70s) for uh running and deploying and

[1:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=70s) for uh running and deploying and training open source models.

[1:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=72s) training open source models.

[1:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=72s) training open source models. Hey guys, my name is Omargari. So um

[1:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=75s) Hey guys, my name is Omargari. So um

[1:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=76s) Hey guys, my name is Omargari. So um usually parents they teach children but

[1:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=78s) usually parents they teach children but

[1:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=78s) usually parents they teach children but Hessen he was teaching me a lot during

[1:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=80s) Hessen he was teaching me a lot during

[1:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=80s) Hessen he was teaching me a lot during the last six years when I uh I got to

[1:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=82s) the last six years when I uh I got to

[1:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=82s) the last six years when I uh I got to coding after 25 years of experience with

[1:25](https://www.youtube.com/watch?v=dznCt8xPW7o&t=85s) coding after 25 years of experience with

[1:25](https://www.youtube.com/watch?v=dznCt8xPW7o&t=85s) coding after 25 years of experience with the big four firms auditing firms uh

[1:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=87s) the big four firms auditing firms uh

[1:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=88s) the big four firms auditing firms uh couple of positions as a CFO and audit

[1:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=90s) couple of positions as a CFO and audit

[1:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=90s) couple of positions as a CFO and audit uh head of internal audit of

[1:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=92s) uh head of internal audit of

[1:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=92s) uh head of internal audit of multi-billion dollar company. So today

[1:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=95s) multi-billion dollar company. So today

[1:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=95s) multi-billion dollar company. So today I'm in the human in the loop for

[1:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=96s) I'm in the human in the loop for

[1:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=96s) I'm in the human in the loop for Hassan's um engineering work.

[1:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=101s) Hassan's um engineering work.

[1:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=101s) Hassan's um engineering work. Okay. So my coding experience didn't

[1:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=104s) Okay. So my coding experience didn't

[1:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=104s) Okay. So my coding experience didn't start just last six years. Um I had a

[1:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=106s) start just last six years. Um I had a

[1:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=106s) start just last six years. Um I had a lot of passion for coding at at age of

[1:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=108s) lot of passion for coding at at age of

[1:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=108s) lot of passion for coding at at age of 16. I was coding in assembly and the

[1:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=111s) 16. I was coding in assembly and the

[1:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=111s) 16. I was coding in assembly and the reason I did that I started with basic a

[1:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=113s) reason I did that I started with basic a

[1:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=113s) reason I did that I started with basic a tennis game small tennis game on an

[1:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=115s) tennis game small tennis game on an

[1:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=115s) tennis game small tennis game on an epson with four lines on the screen and

[1:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=118s) epson with four lines on the screen and

[1:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=118s) epson with four lines on the screen and just this four um you know tennis uh

[2:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=120s) just this four um you know tennis uh

[2:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=120s) just this four um you know tennis uh court but the ball was too slow. It took

[2:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=123s) court but the ball was too slow. It took

[2:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=123s) court but the ball was too slow. It took like almost a minute to get to the other

[2:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=125s) like almost a minute to get to the other

[2:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=125s) like almost a minute to get to the other side. And then I learned assembly

[2:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=127s) side. And then I learned assembly

[2:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=127s) side. And then I learned assembly because of my brother's uh IT

[2:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=128s) because of my brother's uh IT

[2:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=128s) because of my brother's uh IT background. And then I rewrote the the

[2:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=131s) background. And then I rewrote the the

[2:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=131s) background. And then I rewrote the the same game and it took the ball. I can't

[2:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=133s) same game and it took the ball. I can't

[2:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=133s) same game and it took the ball. I can't see it anymore. It was too fast to so I

[2:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=136s) see it anymore. It was too fast to so I

[2:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=136s) see it anymore. It was too fast to so I had to slow it down to really see

[2:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=138s) had to slow it down to really see

[2:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=138s) had to slow it down to really see things. So just wanted to share that

[2:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=139s) things. So just wanted to share that

[2:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=139s) things. So just wanted to share that with you.

[2:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=141s) with you.

[2:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=141s) with you. All right. So spoiler alert. Um the LLM

[2:25](https://www.youtube.com/watch?v=dznCt8xPW7o&t=145s) All right. So spoiler alert. Um the LLM

[2:25](https://www.youtube.com/watch?v=dznCt8xPW7o&t=145s) All right. So spoiler alert. Um the LLM was accurate up until about 4.4%. So the

[2:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=148s) was accurate up until about 4.4%. So the

[2:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=148s) was accurate up until about 4.4%. So the our our forecast of Costco's actual

[2:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=150s) our our forecast of Costco's actual

[2:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=150s) our our forecast of Costco's actual membership was off by 4.4%. And so I I

[2:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=152s) membership was off by 4.4%. And so I I

[2:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=152s) membership was off by 4.4%. And so I I thought this was pretty good, right? We

[2:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=154s) thought this was pretty good, right? We

[2:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=154s) thought this was pretty good, right? We had this LLM build a financial model. We

[2:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=156s) had this LLM build a financial model. We

[2:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=156s) had this LLM build a financial model. We evaled it. Um ended up giving us this

[2:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=158s) evaled it. Um ended up giving us this

[2:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=158s) evaled it. Um ended up giving us this result. Um, and then I showed it to my

[2:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=160s) result. Um, and then I showed it to my

[2:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=160s) result. Um, and then I showed it to my dad and it turned out it got the right

[2:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=162s) dad and it turned out it got the right

[2:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=162s) dad and it turned out it got the right answer or got close to the right answer,

[2:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=164s) answer or got close to the right answer,

[2:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=164s) answer or got close to the right answer, uh, but for the wrong reasons. And so

[2:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=165s) uh, but for the wrong reasons. And so

[2:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=166s) uh, but for the wrong reasons. And so the rest of this talk is going to be

[2:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=167s) the rest of this talk is going to be

[2:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=167s) the rest of this talk is going to be going over, you know, how we built this

[2:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=168s) going over, you know, how we built this

[2:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=168s) going over, you know, how we built this financial model, how we evalued it, and

[2:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=170s) financial model, how we evalued it, and

[2:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=170s) financial model, how we evalued it, and actually what it ended up getting wrong

[2:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=172s) actually what it ended up getting wrong

[2:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=172s) actually what it ended up getting wrong and and kind of lessons learned along

[2:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=174s) and and kind of lessons learned along

[2:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=174s) and and kind of lessons learned along the way, including kind of the

[2:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=175s) the way, including kind of the

[2:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=175s) the way, including kind of the overarching lesson, which is having like

[2:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=177s) overarching lesson, which is having like

[2:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=177s) overarching lesson, which is having like an expert human in the loop, is is very,

[2:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=179s) an expert human in the loop, is is very,

[2:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=179s) an expert human in the loop, is is very, very important when you're building, you

[3:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=181s) very important when you're building, you

[3:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=181s) very important when you're building, you know, AI agents in in these different

[3:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=183s) know, AI agents in in these different

[3:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=183s) know, AI agents in in these different spaces. So, really quick background

[3:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=185s) spaces. So, really quick background

[3:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=185s) spaces. So, really quick background about Costco. Costco sells a lot of

[3:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=187s) about Costco. Costco sells a lot of

[3:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=187s) about Costco. Costco sells a lot of stuff. They sell $300 billion worth of

[3:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=189s) stuff. They sell $300 billion worth of

[3:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=189s) stuff. They sell $300 billion worth of stuff every year at close to cost. So

[3:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=191s) stuff every year at close to cost. So

[3:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=191s) stuff every year at close to cost. So they make very little money from that

[3:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=193s) they make very little money from that

[3:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=193s) they make very little money from that $300 billion. Um actually more than half

[3:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=195s) $300 billion. Um actually more than half

[3:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=195s) $300 billion. Um actually more than half of their revenue is from their

[3:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=197s) of their revenue is from their

[3:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=197s) of their revenue is from their membership fees, right? So they charge

[3:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=198s) membership fees, right? So they charge

[3:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=198s) membership fees, right? So they charge you to to come at the door. Um their

[3:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=201s) you to to come at the door. Um their

[3:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=201s) you to to come at the door. Um their total operating profit was about $12

[3:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=203s) total operating profit was about $12

[3:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=203s) total operating profit was about $12 billion and and a little bit more than

[3:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=204s) billion and and a little bit more than

[3:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=204s) billion and and a little bit more than half of that was from membership fees.

[3:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=206s) half of that was from membership fees.

[3:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=206s) half of that was from membership fees. So membership fees are are, you know,

[3:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=208s) So membership fees are are, you know,

[3:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=208s) So membership fees are are, you know, extremely important when we're trying to

[3:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=209s) extremely important when we're trying to

[3:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=209s) extremely important when we're trying to forecast uh Costco's uh revenue and

[3:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=212s) forecast uh Costco's uh revenue and

[3:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=212s) forecast uh Costco's uh revenue and profits. And thankfully Costco gave us

[3:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=214s) profits. And thankfully Costco gave us

[3:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=214s) profits. And thankfully Costco gave us the perfect test on July 2024 because

[3:37](https://www.youtube.com/watch?v=dznCt8xPW7o&t=217s) the perfect test on July 2024 because

[3:37](https://www.youtube.com/watch?v=dznCt8xPW7o&t=217s) the perfect test on July 2024 because they raised their membership prices for

[3:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=219s) they raised their membership prices for

[3:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=219s) they raised their membership prices for the first time since 2017. So Costco has

[3:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=222s) the first time since 2017. So Costco has

[3:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=222s) the first time since 2017. So Costco has two tiers. They have a standard tier

[3:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=223s) two tiers. They have a standard tier

[3:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=223s) two tiers. They have a standard tier that they raised from 60 to $65 and they

[3:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=226s) that they raised from 60 to $65 and they

[3:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=226s) that they raised from 60 to $65 and they have an executive tier that they raised

[3:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=227s) have an executive tier that they raised

[3:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=228s) have an executive tier that they raised from 120 to $130 uh. So what we did is

[3:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=231s) from 120 to $130 uh. So what we did is

[3:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=231s) from 120 to $130 uh. So what we did is we came up with this cutoff which was a

[3:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=233s) we came up with this cutoff which was a

[3:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=233s) we came up with this cutoff which was a couple weeks before the announcement and

[3:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=235s) couple weeks before the announcement and

[3:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=235s) couple weeks before the announcement and we took an open source model and we gave

[3:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=238s) we took an open source model and we gave

[3:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=238s) we took an open source model and we gave it access to all of these documents from

[4:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=240s) it access to all of these documents from

[4:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=240s) it access to all of these documents from Costco uh up until that point and asked

[4:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=243s) Costco uh up until that point and asked

[4:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=243s) Costco uh up until that point and asked it like hey if Costco you know increases

[4:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=245s) it like hey if Costco you know increases

[4:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=245s) it like hey if Costco you know increases its membership fee um what are the the

[4:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=249s) its membership fee um what are the the

[4:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=249s) its membership fee um what are the the the revenue and the profit that you

[4:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=251s) the revenue and the profit that you

[4:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=251s) the revenue and the profit that you would uh forecast. So we actually

[4:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=252s) would uh forecast. So we actually

[4:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=252s) would uh forecast. So we actually started doing this with a couple of

[4:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=254s) started doing this with a couple of

[4:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=254s) started doing this with a couple of recent open source models like GLM 5.3

[4:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=256s) recent open source models like GLM 5.3

[4:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=256s) recent open source models like GLM 5.3 and Kimmy K3. But it turns out they

[4:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=258s) and Kimmy K3. But it turns out they

[4:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=258s) and Kimmy K3. But it turns out they already knew that Costco actually

[4:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=260s) already knew that Costco actually

[4:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=260s) already knew that Costco actually increased their prices um because it was

[4:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=262s) increased their prices um because it was

[4:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=262s) increased their prices um because it was already in their training data, right?

[4:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=263s) already in their training data, right?

[4:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=264s) already in their training data, right? So we had to go and find, you know,

[4:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=266s) So we had to go and find, you know,

[4:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=266s) So we had to go and find, you know, older open models who were trained, you

[4:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=268s) older open models who were trained, you

[4:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=268s) older open models who were trained, you know, up until before July 2024. So we

[4:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=271s) know, up until before July 2024. So we

[4:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=271s) know, up until before July 2024. So we found GBT OSS120B and Llama 3.3 uh7B. Uh

[4:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=275s) found GBT OSS120B and Llama 3.3 uh7B. Uh

[4:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=275s) found GBT OSS120B and Llama 3.3 uh7B. Uh and we this is an example of a question

[4:37](https://www.youtube.com/watch?v=dznCt8xPW7o&t=277s) and we this is an example of a question

[4:37](https://www.youtube.com/watch?v=dznCt8xPW7o&t=277s) and we this is an example of a question we asked it. Hey, did Costco change its

[4:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=279s) we asked it. Hey, did Costco change its

[4:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=279s) we asked it. Hey, did Costco change its membership fees in 2024? The newer

[4:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=281s) membership fees in 2024? The newer

[4:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=281s) membership fees in 2024? The newer models uh knew the answer. The older

[4:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=283s) models uh knew the answer. The older

[4:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=283s) models uh knew the answer. The older models didn't. And so we wanted to give

[4:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=285s) models didn't. And so we wanted to give

[4:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=285s) models didn't. And so we wanted to give we wanted to use these older models

[4:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=286s) we wanted to use these older models

[4:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=286s) we wanted to use these older models because you know you want a completely

[4:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=288s) because you know you want a completely

[4:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=288s) because you know you want a completely unbiased uh uh point of view here. So

[4:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=290s) unbiased uh uh point of view here. So

[4:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=290s) unbiased uh uh point of view here. So that's kind of the first lesson. If

[4:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=291s) that's kind of the first lesson. If

[4:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=291s) that's kind of the first lesson. If you're trying to do any sort of

[4:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=292s) you're trying to do any sort of

[4:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=292s) you're trying to do any sort of forecasting, you need to try to find a

[4:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=293s) forecasting, you need to try to find a

[4:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=293s) forecasting, you need to try to find a model whose training data ends before

[4:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=296s) model whose training data ends before

[4:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=296s) model whose training data ends before the event that you're actually trying to

[4:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=297s) the event that you're actually trying to

[4:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=297s) the event that you're actually trying to forecast. Um and uh for you know open

[5:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=300s) forecast. Um and uh for you know open

[5:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=300s) forecast. Um and uh for you know open weight models are actually really good

[5:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=301s) weight models are actually really good

[5:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=301s) weight models are actually really good for for this kind of use case. Um so

[5:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=304s) for for this kind of use case. Um so

[5:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=304s) for for this kind of use case. Um so this is kind of how everything works. We

[5:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=306s) this is kind of how everything works. We

[5:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=306s) this is kind of how everything works. We went and scraped about 214 uh public

[5:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=308s) went and scraped about 214 uh public

[5:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=308s) went and scraped about 214 uh public filings from Costco. uh you know,

[5:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=311s) filings from Costco. uh you know,

[5:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=311s) filings from Costco. uh you know, quarterly and yearly earning reports,

[5:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=312s) quarterly and yearly earning reports,

[5:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=312s) quarterly and yearly earning reports, sales reports, um all of that. We uh

[5:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=316s) sales reports, um all of that. We uh

[5:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=316s) sales reports, um all of that. We uh ended up making it into a list of about

[5:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=318s) ended up making it into a list of about

[5:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=318s) ended up making it into a list of about 3,000 facts that were all timestamped

[5:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=320s) 3,000 facts that were all timestamped

[5:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=320s) 3,000 facts that were all timestamped and and then we filtered these facts to

[5:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=323s) and and then we filtered these facts to

[5:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=323s) and and then we filtered these facts to only the facts available before our

[5:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=324s) only the facts available before our

[5:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=324s) only the facts available before our cutoff date. We gave it to a model and

[5:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=326s) cutoff date. We gave it to a model and

[5:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=326s) cutoff date. We gave it to a model and we actually spun up um 16 sessions of of

[5:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=329s) we actually spun up um 16 sessions of of

[5:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=329s) we actually spun up um 16 sessions of of GPTOSs 12B. We had the model actually

[5:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=333s) GPTOSs 12B. We had the model actually

[5:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=333s) GPTOSs 12B. We had the model actually write a bunch of formulas and write this

[5:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=334s) write a bunch of formulas and write this

[5:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=334s) write a bunch of formulas and write this financial model. we ran some checks and

[5:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=336s) financial model. we ran some checks and

[5:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=336s) financial model. we ran some checks and then we picked uh the from those 16 runs

[5:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=339s) then we picked uh the from those 16 runs

[5:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=339s) then we picked uh the from those 16 runs we picked the ones that actually passed

[5:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=341s) we picked the ones that actually passed

[5:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=341s) we picked the ones that actually passed uh our checks. So in in this case like

[5:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=343s) uh our checks. So in in this case like

[5:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=344s) uh our checks. So in in this case like what does the model actually do right in

[5:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=345s) what does the model actually do right in

[5:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=345s) what does the model actually do right in this previous slide where we we gave it

[5:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=347s) this previous slide where we we gave it

[5:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=347s) this previous slide where we we gave it all these facts. What does the model

[5:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=349s) all these facts. What does the model

[5:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=349s) all these facts. What does the model output here? So we actually had the

[5:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=350s) output here? So we actually had the

[5:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=350s) output here? So we actually had the model output formulas here um in and and

[5:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=354s) model output formulas here um in and and

[5:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=354s) model output formulas here um in and and so an example of this is all the way to

[5:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=356s) so an example of this is all the way to

[5:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=356s) so an example of this is all the way to the left we say like we account for

[5:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=358s) the left we say like we account for

[5:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=358s) the left we say like we account for membership fee revenue on a deferred

[5:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=359s) membership fee revenue on a deferred

[5:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=359s) membership fee revenue on a deferred basis. This is straight from Costco's

[6:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=361s) basis. This is straight from Costco's

[6:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=361s) basis. This is straight from Costco's quarterly report. We give that to the

[6:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=362s) quarterly report. We give that to the

[6:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=362s) quarterly report. We give that to the model and it constructs this uh uh

[6:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=364s) model and it constructs this uh uh

[6:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=364s) model and it constructs this uh uh structure JSON where it says you know

[6:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=366s) structure JSON where it says you know

[6:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=366s) structure JSON where it says you know the operand is ratable the term is 52

[6:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=368s) the operand is ratable the term is 52

[6:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=368s) the operand is ratable the term is 52 weeks. And so what this means is

[6:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=370s) weeks. And so what this means is

[6:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=370s) weeks. And so what this means is basically, you know, even if a user

[6:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=371s) basically, you know, even if a user

[6:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=372s) basically, you know, even if a user comes in and pays a $65 membership fee,

[6:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=374s) comes in and pays a $65 membership fee,

[6:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=374s) comes in and pays a $65 membership fee, it's actually spread across 52 weeks.

[6:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=376s) it's actually spread across 52 weeks.

[6:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=376s) it's actually spread across 52 weeks. Um, and so this is really really

[6:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=378s) Um, and so this is really really

[6:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=378s) Um, and so this is really really important when you're doing uh uh uh

[6:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=379s) important when you're doing uh uh uh

[6:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=379s) important when you're doing uh uh uh forecasting to understand how to to to

[6:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=382s) forecasting to understand how to to to

[6:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=382s) forecasting to understand how to to to forecast. And so we had the model

[6:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=384s) forecast. And so we had the model

[6:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=384s) forecast. And so we had the model construct a list of all of these

[6:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=386s) construct a list of all of these

[6:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=386s) construct a list of all of these formulas. And so the main takeaway here

[6:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=387s) formulas. And so the main takeaway here

[6:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=388s) formulas. And so the main takeaway here is like have models write structured

[6:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=389s) is like have models write structured

[6:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=389s) is like have models write structured data instead of um having them, you

[6:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=392s) data instead of um having them, you

[6:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=392s) data instead of um having them, you know, do the math and do and do

[6:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=393s) know, do the math and do and do

[6:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=393s) know, do the math and do and do everything all at once. Um the model

[6:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=395s) everything all at once. Um the model

[6:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=395s) everything all at once. Um the model also came up with a bunch of

[6:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=396s) also came up with a bunch of

[6:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=396s) also came up with a bunch of assumptions. Uh and so it had to guess

[6:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=398s) assumptions. Uh and so it had to guess

[6:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=398s) assumptions. Uh and so it had to guess on a few things. So, for example, what

[6:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=399s) on a few things. So, for example, what

[6:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=400s) on a few things. So, for example, what percentage of people are in the standard

[6:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=401s) percentage of people are in the standard

[6:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=401s) percentage of people are in the standard tier versus the executive tier? Um, how

[6:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=404s) tier versus the executive tier? Um, how

[6:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=404s) tier versus the executive tier? Um, how many members were in Canada can US and

[6:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=406s) many members were in Canada can US and

[6:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=406s) many members were in Canada can US and Canada and impacted by this change. U,

[6:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=408s) Canada and impacted by this change. U,

[6:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=408s) Canada and impacted by this change. U, so it made a few assumptions. My dad's

[6:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=410s) so it made a few assumptions. My dad's

[6:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=410s) so it made a few assumptions. My dad's going to go over some of these

[6:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=410s) going to go over some of these

[6:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=410s) going to go over some of these assumptions later and why some of them

[6:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=412s) assumptions later and why some of them

[6:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=412s) assumptions later and why some of them may or may not be wrong. Um, and and

[6:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=414s) may or may not be wrong. Um, and and

[6:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=414s) may or may not be wrong. Um, and and here's kind of the full picture over

[6:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=415s) here's kind of the full picture over

[6:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=415s) here's kind of the full picture over here for the full financial model. On

[6:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=417s) here for the full financial model. On

[6:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=417s) here for the full financial model. On the left, you have the assumptions,

[6:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=418s) the left, you have the assumptions,

[6:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=418s) the left, you have the assumptions, which we went over some on the last

[6:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=419s) which we went over some on the last

[7:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=420s) which we went over some on the last slide, um, that the model came up with.

[7:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=422s) slide, um, that the model came up with.

[7:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=422s) slide, um, that the model came up with. And then we have 14 formulas that the

[7:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=424s) And then we have 14 formulas that the

[7:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=424s) And then we have 14 formulas that the model also came up with. Um, and and

[7:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=426s) model also came up with. Um, and and

[7:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=426s) model also came up with. Um, and and then we have a few numbers that are kind

[7:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=427s) then we have a few numbers that are kind

[7:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=427s) then we have a few numbers that are kind of directly from uh Costco's filings.

[7:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=429s) of directly from uh Costco's filings.

[7:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=429s) of directly from uh Costco's filings. And this is actually an Excel

[7:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=431s) And this is actually an Excel

[7:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=431s) And this is actually an Excel spreadsheet that that I created. I'm

[7:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=432s) spreadsheet that that I created. I'm

[7:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=432s) spreadsheet that that I created. I'm kind of just trying to visualize it in a

[7:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=433s) kind of just trying to visualize it in a

[7:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=434s) kind of just trying to visualize it in a way uh where where it's uh it's a little

[7:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=436s) way uh where where it's uh it's a little

[7:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=436s) way uh where where it's uh it's a little bit nicer to to look at. Um, and so

[7:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=438s) bit nicer to to look at. Um, and so

[7:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=438s) bit nicer to to look at. Um, and so after we had the model create this

[7:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=440s) after we had the model create this

[7:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=440s) after we had the model create this financial model, we uh actually had it

[7:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=444s) financial model, we uh actually had it

[7:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=444s) financial model, we uh actually had it go through a bunch of these checks. Uh,

[7:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=446s) go through a bunch of these checks. Uh,

[7:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=446s) go through a bunch of these checks. Uh, and so there's just basic checks. For

[7:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=447s) and so there's just basic checks. For

[7:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=447s) and so there's just basic checks. For example, a formula can't depend on

[7:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=449s) example, a formula can't depend on

[7:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=449s) example, a formula can't depend on itself. For example, raising the

[7:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=450s) itself. For example, raising the

[7:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=450s) itself. For example, raising the membership fee needs to raise revenue.

[7:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=452s) membership fee needs to raise revenue.

[7:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=452s) membership fee needs to raise revenue. you know, just making sure that like

[7:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=453s) you know, just making sure that like

[7:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=453s) you know, just making sure that like some a few basic things were were

[7:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=455s) some a few basic things were were

[7:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=455s) some a few basic things were were correct. And whenever a model whenever

[7:37](https://www.youtube.com/watch?v=dznCt8xPW7o&t=457s) correct. And whenever a model whenever

[7:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=458s) correct. And whenever a model whenever one of those runs, because remember we

[7:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=459s) one of those runs, because remember we

[7:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=459s) one of those runs, because remember we had these 16 runs, whenever one of those

[7:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=461s) had these 16 runs, whenever one of those

[7:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=461s) had these 16 runs, whenever one of those runs got one of these things wrong, we

[7:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=462s) runs got one of these things wrong, we

[7:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=462s) runs got one of these things wrong, we ended up passing the error back to the

[7:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=464s) ended up passing the error back to the

[7:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=464s) ended up passing the error back to the model uh up to two different times uh so

[7:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=467s) model uh up to two different times uh so

[7:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=467s) model uh up to two different times uh so that it kind of tried to self-correct

[7:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=469s) that it kind of tried to self-correct

[7:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=469s) that it kind of tried to self-correct itself. And what was interesting here,

[7:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=471s) itself. And what was interesting here,

[7:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=471s) itself. And what was interesting here, so these are actually the the 16

[7:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=473s) so these are actually the the 16

[7:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=473s) so these are actually the the 16 different runs that we have. They're all

[7:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=474s) different runs that we have. They're all

[7:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=474s) different runs that we have. They're all running. They're all creating their

[7:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=475s) running. They're all creating their

[7:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=475s) running. They're all creating their financial model. They're writing the the

[7:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=477s) financial model. They're writing the the

[7:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=477s) financial model. They're writing the the formulas. We're passing the errors back.

[7:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=479s) formulas. We're passing the errors back.

[7:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=479s) formulas. We're passing the errors back. And what ended up happening is something

[8:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=481s) And what ended up happening is something

[8:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=481s) And what ended up happening is something interesting where we had a bunch of them

[8:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=482s) interesting where we had a bunch of them

[8:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=482s) interesting where we had a bunch of them fail anyway. Um because they just failed

[8:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=485s) fail anyway. Um because they just failed

[8:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=485s) fail anyway. Um because they just failed more than the two times we we we gave

[8:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=487s) more than the two times we we we gave

[8:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=487s) more than the two times we we we gave them back their their error. Um and so

[8:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=489s) them back their their error. Um and so

[8:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=489s) them back their their error. Um and so we ended up uh using these five that

[8:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=491s) we ended up uh using these five that

[8:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=491s) we ended up uh using these five that ended up passing uh for for our uh

[8:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=493s) ended up passing uh for for our uh

[8:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=493s) ended up passing uh for for our uh financial forecast in general. And

[8:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=495s) financial forecast in general. And

[8:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=495s) financial forecast in general. And that's kind of another lesson. We ended

[8:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=497s) that's kind of another lesson. We ended

[8:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=497s) that's kind of another lesson. We ended up starting with just like one prompt

[8:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=498s) up starting with just like one prompt

[8:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=498s) up starting with just like one prompt that we kept tuning and tuning and

[8:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=499s) that we kept tuning and tuning and

[8:19](https://www.youtube.com/watch?v=dznCt8xPW7o&t=499s) that we kept tuning and tuning and tuning. We're like, "Oh, like surely

[8:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=501s) tuning. We're like, "Oh, like surely

[8:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=501s) tuning. We're like, "Oh, like surely like we're going to give this a GPT

[8:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=502s) like we're going to give this a GPT

[8:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=502s) like we're going to give this a GPT OSS120B and it's going to pass, you

[8:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=504s) OSS120B and it's going to pass, you

[8:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=504s) OSS120B and it's going to pass, you know, all the checks." Um, but that

[8:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=506s) know, all the checks." Um, but that

[8:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=506s) know, all the checks." Um, but that ended up like just taking so much time

[8:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=507s) ended up like just taking so much time

[8:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=507s) ended up like just taking so much time that we were like, what if we just spun

[8:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=509s) that we were like, what if we just spun

[8:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=509s) that we were like, what if we just spun up, you know, 20 16 different runs. Uh,

[8:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=512s) up, you know, 20 16 different runs. Uh,

[8:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=512s) up, you know, 20 16 different runs. Uh, and in this case, it works really well

[8:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=513s) and in this case, it works really well

[8:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=513s) and in this case, it works really well because the model is really cheap. So,

[8:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=514s) because the model is really cheap. So,

[8:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=514s) because the model is really cheap. So, it only cost us 19 cents to spin up

[8:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=516s) it only cost us 19 cents to spin up

[8:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=516s) it only cost us 19 cents to spin up these like 16 different runs and we had

[8:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=518s) these like 16 different runs and we had

[8:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=518s) these like 16 different runs and we had more of them that that passed this way.

[8:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=521s) more of them that that passed this way.

[8:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=521s) more of them that that passed this way. So, this is the million-dollar question

[8:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=523s) So, this is the million-dollar question

[8:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=523s) So, this is the million-dollar question that we asked, right? If cost what if

[8:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=525s) that we asked, right? If cost what if

[8:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=525s) that we asked, right? If cost what if Costco raises its fee uh this September,

[8:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=528s) Costco raises its fee uh this September,

[8:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=528s) Costco raises its fee uh this September, what happens to its actual revenue? So

[8:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=530s) what happens to its actual revenue? So

[8:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=530s) what happens to its actual revenue? So we had GLM 5.3 translated into just like

[8:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=533s) we had GLM 5.3 translated into just like

[8:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=533s) we had GLM 5.3 translated into just like some structured output that we can feed

[8:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=534s) some structured output that we can feed

[8:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=534s) some structured output that we can feed into the financial model. And then the

[8:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=536s) into the financial model. And then the

[8:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=536s) into the financial model. And then the model gave us this number of like it

[8:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=538s) model gave us this number of like it

[8:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=538s) model gave us this number of like it predicted that uh profit would increase

[9:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=541s) predicted that uh profit would increase

[9:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=541s) predicted that uh profit would increase by $560 million over two years. Uh and

[9:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=545s) by $560 million over two years. Uh and

[9:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=545s) by $560 million over two years. Uh and so this is it just in the app. We send

[9:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=547s) so this is it just in the app. We send

[9:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=547s) so this is it just in the app. We send this over uh GLM 5.3 interprets it. It

[9:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=550s) this over uh GLM 5.3 interprets it. It

[9:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=550s) this over uh GLM 5.3 interprets it. It sends it as structured data to the

[9:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=551s) sends it as structured data to the

[9:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=551s) sends it as structured data to the model. And remember the model is just a

[9:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=553s) model. And remember the model is just a

[9:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=553s) model. And remember the model is just a financial model. It takes 10

[9:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=554s) financial model. It takes 10

[9:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=554s) financial model. It takes 10 milliseconds to run uh because the mo

[9:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=556s) milliseconds to run uh because the mo

[9:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=556s) milliseconds to run uh because the mo because the uh our GPOSS model already

[9:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=558s) because the uh our GPOSS model already

[9:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=558s) because the uh our GPOSS model already made it. And here's that 560 million uh

[9:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=561s) made it. And here's that 560 million uh

[9:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=561s) made it. And here's that 560 million uh number right there. So, um this is it

[9:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=563s) number right there. So, um this is it

[9:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=563s) number right there. So, um this is it just like visualized on a scale. The

[9:25](https://www.youtube.com/watch?v=dznCt8xPW7o&t=565s) just like visualized on a scale. The

[9:25](https://www.youtube.com/watch?v=dznCt8xPW7o&t=565s) just like visualized on a scale. The blue line is the predicted uh uh uh

[9:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=568s) blue line is the predicted uh uh uh

[9:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=568s) blue line is the predicted uh uh uh profit and orange is actually the real

[9:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=571s) profit and orange is actually the real

[9:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=571s) profit and orange is actually the real profit extracted from Costco's uh

[9:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=573s) profit extracted from Costco's uh

[9:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=573s) profit extracted from Costco's uh earning reports. And so you can see we

[9:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=574s) earning reports. And so you can see we

[9:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=574s) earning reports. And so you can see we were accurate to about 4.4% uh which is

[9:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=578s) were accurate to about 4.4% uh which is

[9:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=578s) were accurate to about 4.4% uh which is the average of the five builds. The

[9:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=579s) the average of the five builds. The

[9:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=579s) the average of the five builds. The builds gave us between 0.7 and 5.2%. So

[9:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=583s) builds gave us between 0.7 and 5.2%. So

[9:43](https://www.youtube.com/watch?v=dznCt8xPW7o&t=583s) builds gave us between 0.7 and 5.2%. So again, I thought this was pretty good,

[9:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=584s) again, I thought this was pretty good,

[9:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=584s) again, I thought this was pretty good, right? We had the model build uh we had

[9:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=586s) right? We had the model build uh we had

[9:46](https://www.youtube.com/watch?v=dznCt8xPW7o&t=586s) right? We had the model build uh we had our AI model build this financial model.

[9:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=588s) our AI model build this financial model.

[9:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=588s) our AI model build this financial model. We ran it. It was accurate within 4.4%.

[9:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=591s) We ran it. It was accurate within 4.4%.

[9:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=591s) We ran it. It was accurate within 4.4%. Uh and then I showed it to my dad. Yeah,

[9:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=594s) Uh and then I showed it to my dad. Yeah,

[9:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=594s) Uh and then I showed it to my dad. Yeah, definitely. It's amazing to see that AI

[9:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=595s) definitely. It's amazing to see that AI

[9:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=596s) definitely. It's amazing to see that AI can can get to financial models which is

[9:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=598s) can can get to financial models which is

[9:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=598s) can can get to financial models which is the you know the area of accountants and

[9:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=599s) the you know the area of accountants and

[9:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=599s) the you know the area of accountants and and financial people but with with very

[10:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=602s) and financial people but with with very

[10:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=602s) and financial people but with with very small guidance but the first thing you

[10:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=604s) small guidance but the first thing you

[10:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=604s) small guidance but the first thing you look at when you try to look at a

[10:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=606s) look at when you try to look at a

[10:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=606s) look at when you try to look at a financial model is to audit it is to

[10:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=608s) financial model is to audit it is to

[10:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=608s) financial model is to audit it is to look at assumptions. So these are some

[10:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=610s) look at assumptions. So these are some

[10:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=610s) look at assumptions. So these are some of the assumptions that were used in

[10:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=611s) of the assumptions that were used in

[10:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=611s) of the assumptions that were used in Hassan's model and uh I just want to

[10:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=614s) Hassan's model and uh I just want to

[10:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=614s) Hassan's model and uh I just want to talk your attention to two things. The

[10:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=615s) talk your attention to two things. The

[10:15](https://www.youtube.com/watch?v=dznCt8xPW7o&t=615s) talk your attention to two things. The executive share like Hassan mentioned

[10:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=617s) executive share like Hassan mentioned

[10:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=617s) executive share like Hassan mentioned before. Uh it was um you know big swings

[10:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=620s) before. Uh it was um you know big swings

[10:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=620s) before. Uh it was um you know big swings there. Um US Canada the last um you know

[10:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=623s) there. Um US Canada the last um you know

[10:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=623s) there. Um US Canada the last um you know um assumption did not move um sales and

[10:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=627s) um assumption did not move um sales and

[10:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=627s) um assumption did not move um sales and um and fees with much with zero. So

[10:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=631s) um and fees with much with zero. So

[10:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=631s) um and fees with much with zero. So basically that's a lever that is not

[10:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=632s) basically that's a lever that is not

[10:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=632s) basically that's a lever that is not working and it should be and something

[10:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=634s) working and it should be and something

[10:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=634s) working and it should be and something was wrong. Something went wrong there

[10:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=636s) was wrong. Something went wrong there

[10:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=636s) was wrong. Something went wrong there and it should be fixed.

[10:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=639s) and it should be fixed.

[10:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=639s) and it should be fixed. Um so these are why the executive share

[10:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=642s) Um so these are why the executive share

[10:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=642s) Um so these are why the executive share for example the executive share the 15%

[10:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=644s) for example the executive share the 15%

[10:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=644s) for example the executive share the 15% the mother took 15%. Um and the problem

[10:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=647s) the mother took 15%. Um and the problem

[10:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=647s) the mother took 15%. Um and the problem is that the mother had just one citation

[10:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=650s) is that the mother had just one citation

[10:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=650s) is that the mother had just one citation increased penetration of executive

[10:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=651s) increased penetration of executive

[10:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=651s) increased penetration of executive membership but there was no number to

[10:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=654s) membership but there was no number to

[10:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=654s) membership but there was no number to back it up. So that's what the auditors

[10:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=656s) back it up. So that's what the auditors

[10:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=656s) back it up. So that's what the auditors looking at at how are all the facts

[10:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=658s) looking at at how are all the facts

[10:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=658s) looking at at how are all the facts backed up with you know with with real

[11:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=660s) backed up with you know with with real

[11:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=660s) backed up with you know with with real numbers. So the citation is real but the

[11:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=663s) numbers. So the citation is real but the

[11:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=663s) numbers. So the citation is real but the problem and the the the number was ma

[11:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=665s) problem and the the the number was ma

[11:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=665s) problem and the the the number was ma was made up by AI and not only that AI

[11:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=668s) was made up by AI and not only that AI

[11:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=668s) was made up by AI and not only that AI also draw the boundaries of that

[11:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=670s) also draw the boundaries of that

[11:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=670s) also draw the boundaries of that assumption which is uh zero to um 0 to 5

[11:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=674s) assumption which is uh zero to um 0 to 5

[11:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=674s) assumption which is uh zero to um 0 to 5 to 30%. But we will see that it's not

[11:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=677s) to 30%. But we will see that it's not

[11:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=677s) to 30%. But we will see that it's not correct actually um the that's what the

[11:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=680s) correct actually um the that's what the

[11:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=680s) correct actually um the that's what the AI did 5 to 30% but the problem is the

[11:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=684s) AI did 5 to 30% but the problem is the

[11:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=684s) AI did 5 to 30% but the problem is the real number that was already disclosed

[11:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=686s) real number that was already disclosed

[11:26](https://www.youtube.com/watch?v=dznCt8xPW7o&t=686s) real number that was already disclosed in the in in the filings um was 45%. So

[11:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=690s) in the in in the filings um was 45%. So

[11:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=690s) in the in in the filings um was 45%. So in this case AI failed to um you know um

[11:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=694s) in this case AI failed to um you know um

[11:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=694s) in this case AI failed to um you know um identify that number from the from the

[11:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=696s) identify that number from the from the

[11:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=696s) identify that number from the from the filing. So that's one uh one problem and

[11:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=699s) filing. So that's one uh one problem and

[11:39](https://www.youtube.com/watch?v=dznCt8xPW7o&t=699s) filing. So that's one uh one problem and um the surprise why did it land correct

[11:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=702s) um the surprise why did it land correct

[11:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=702s) um the surprise why did it land correct like hasn't mentioned it's 4.4% 4% which

[11:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=705s) like hasn't mentioned it's 4.4% 4% which

[11:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=705s) like hasn't mentioned it's 4.4% 4% which is a okay uh performance because of two

[11:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=710s) is a okay uh performance because of two

[11:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=710s) is a okay uh performance because of two errors that canceled out two assumptions

[11:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=712s) errors that canceled out two assumptions

[11:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=712s) errors that canceled out two assumptions that canceled out and the first one was

[11:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=714s) that canceled out and the first one was

[11:54](https://www.youtube.com/watch?v=dznCt8xPW7o&t=714s) that canceled out and the first one was an overstatement of the base how many

[11:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=717s) an overstatement of the base how many

[11:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=717s) an overstatement of the base how many people instead of applying the um

[11:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=719s) people instead of applying the um

[11:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=719s) people instead of applying the um increase on just the US Canada portion

[12:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=722s) increase on just the US Canada portion

[12:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=722s) increase on just the US Canada portion of the members the model applied it to

[12:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=725s) of the members the model applied it to

[12:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=725s) of the members the model applied it to international to to the whole population

[12:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=727s) international to to the whole population

[12:07](https://www.youtube.com/watch?v=dznCt8xPW7o&t=727s) international to to the whole population actually and the second thing that was

[12:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=729s) actually and the second thing that was

[12:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=729s) actually and the second thing that was understated it was too low it's the

[12:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=731s) understated it was too low it's the

[12:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=731s) understated it was too low it's the percentage uh instead of 45% which is

[12:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=734s) percentage uh instead of 45% which is

[12:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=734s) percentage uh instead of 45% which is the real number. Um AI made up this 15%

[12:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=737s) the real number. Um AI made up this 15%

[12:17](https://www.youtube.com/watch?v=dznCt8xPW7o&t=737s) the real number. Um AI made up this 15% and that was too low. It landed $69

[12:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=741s) and that was too low. It landed $69

[12:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=741s) and that was too low. It landed $69 which is close to 66 but um again the

[12:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=743s) which is close to 66 but um again the

[12:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=744s) which is close to 66 but um again the model behind it was wrong. And this is

[12:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=747s) model behind it was wrong. And this is

[12:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=747s) model behind it was wrong. And this is um this is what Hassan was mentioning

[12:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=749s) um this is what Hassan was mentioning

[12:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=749s) um this is what Hassan was mentioning before um that um like that led to um

[12:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=754s) before um that um like that led to um

[12:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=754s) before um that um like that led to um like the 16 the 16 models they failed

[12:37](https://www.youtube.com/watch?v=dznCt8xPW7o&t=757s) like the 16 the 16 models they failed

[12:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=758s) like the 16 the 16 models they failed and they failed um all with the same

[12:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=760s) and they failed um all with the same

[12:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=760s) and they failed um all with the same thing with big ranges from half billion

[12:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=762s) thing with big ranges from half billion

[12:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=762s) thing with big ranges from half billion to $6 billion in in terms of fee

[12:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=765s) to $6 billion in in terms of fee

[12:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=765s) to $6 billion in in terms of fee increase while the actual number that we

[12:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=768s) increase while the actual number that we

[12:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=768s) increase while the actual number that we recalculated was just $450 million. So

[12:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=771s) recalculated was just $450 million. So

[12:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=771s) recalculated was just $450 million. So that that was one of the other issues

[12:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=773s) that that was one of the other issues

[12:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=773s) that that was one of the other issues here. And then um just a trick here the

[12:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=776s) here. And then um just a trick here the

[12:56](https://www.youtube.com/watch?v=dznCt8xPW7o&t=776s) here. And then um just a trick here the way to know whether a model or um to to

[13:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=780s) way to know whether a model or um to to

[13:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=780s) way to know whether a model or um to to something is wrong if you have two

[13:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=782s) something is wrong if you have two

[13:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=782s) something is wrong if you have two things cancelling out like an auditing

[13:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=784s) things cancelling out like an auditing

[13:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=784s) things cancelling out like an auditing trick you just fix one of the two and

[13:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=786s) trick you just fix one of the two and

[13:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=786s) trick you just fix one of the two and the model will look worse. So before the

[13:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=789s) the model will look worse. So before the

[13:09](https://www.youtube.com/watch?v=dznCt8xPW7o&t=789s) the model will look worse. So before the performance was 1.7 because things were

[13:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=791s) performance was 1.7 because things were

[13:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=791s) performance was 1.7 because things were washing out canceling out and then once

[13:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=794s) washing out canceling out and then once

[13:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=794s) washing out canceling out and then once we correct just one we just we put like

[13:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=796s) we correct just one we just we put like

[13:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=796s) we correct just one we just we put like the 45% instead of 15% that was made up

[13:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=800s) the 45% instead of 15% that was made up

[13:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=800s) the 45% instead of 15% that was made up by AI suddenly the problem the error was

[13:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=803s) by AI suddenly the problem the error was

[13:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=803s) by AI suddenly the problem the error was 28%. So that's what we say luck is not a

[13:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=807s) 28%. So that's what we say luck is not a

[13:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=807s) 28%. So that's what we say luck is not a control in the auditing world. Usually

[13:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=810s) control in the auditing world. Usually

[13:30](https://www.youtube.com/watch?v=dznCt8xPW7o&t=810s) control in the auditing world. Usually in auditing we use three lines of

[13:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=811s) in auditing we use three lines of

[13:31](https://www.youtube.com/watch?v=dznCt8xPW7o&t=811s) in auditing we use three lines of defense. We have management controls

[13:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=813s) defense. We have management controls

[13:33](https://www.youtube.com/watch?v=dznCt8xPW7o&t=813s) defense. We have management controls which is the engineering checks that

[13:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=815s) which is the engineering checks that

[13:35](https://www.youtube.com/watch?v=dznCt8xPW7o&t=815s) which is the engineering checks that hasn't um coded and then we have the uh

[13:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=818s) hasn't um coded and then we have the uh

[13:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=818s) hasn't um coded and then we have the uh internal audit or some some internal um

[13:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=821s) internal audit or some some internal um

[13:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=821s) internal audit or some some internal um accounting checks that should be

[13:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=822s) accounting checks that should be

[13:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=822s) accounting checks that should be performed by internal um uh audit

[13:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=824s) performed by internal um uh audit

[13:44](https://www.youtube.com/watch?v=dznCt8xPW7o&t=824s) performed by internal um uh audit function and then we have a third layer

[13:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=827s) function and then we have a third layer

[13:47](https://www.youtube.com/watch?v=dznCt8xPW7o&t=827s) function and then we have a third layer which is the independent check that's

[13:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=829s) which is the independent check that's

[13:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=829s) which is the independent check that's external auditors. So these layers they

[13:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=831s) external auditors. So these layers they

[13:52](https://www.youtube.com/watch?v=dznCt8xPW7o&t=832s) external auditors. So these layers they will benefit like the model will benefit

[13:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=833s) will benefit like the model will benefit

[13:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=833s) will benefit like the model will benefit from adding layer two and three and

[13:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=835s) from adding layer two and three and

[13:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=835s) from adding layer two and three and that's the role I was playing like this

[13:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=838s) that's the role I was playing like this

[13:58](https://www.youtube.com/watch?v=dznCt8xPW7o&t=838s) that's the role I was playing like this CPA in the loop or what I call uh

[14:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=840s) CPA in the loop or what I call uh

[14:00](https://www.youtube.com/watch?v=dznCt8xPW7o&t=840s) CPA in the loop or what I call uh actually the judgment professional

[14:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=843s) actually the judgment professional

[14:03](https://www.youtube.com/watch?v=dznCt8xPW7o&t=843s) actually the judgment professional judgment in the loop um to because we

[14:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=845s) judgment in the loop um to because we

[14:05](https://www.youtube.com/watch?v=dznCt8xPW7o&t=845s) judgment in the loop um to because we can we can hardcode most of those checks

[14:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=848s) can we can hardcode most of those checks

[14:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=848s) can we can hardcode most of those checks but still we need uh we need a human um

[14:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=851s) but still we need uh we need a human um

[14:11](https://www.youtube.com/watch?v=dznCt8xPW7o&t=851s) but still we need uh we need a human um looking at that.

[14:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=852s) looking at that.

[14:12](https://www.youtube.com/watch?v=dznCt8xPW7o&t=852s) looking at that. Great. So, you know, as my dad said, we

[14:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=854s) Great. So, you know, as my dad said, we

[14:14](https://www.youtube.com/watch?v=dznCt8xPW7o&t=854s) Great. So, you know, as my dad said, we we had the model here. We passed it

[14:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=856s) we had the model here. We passed it

[14:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=856s) we had the model here. We passed it through engineering checks. It did

[14:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=858s) through engineering checks. It did

[14:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=858s) through engineering checks. It did pretty well. Uh, you know, we got a

[14:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=860s) pretty well. Uh, you know, we got a

[14:20](https://www.youtube.com/watch?v=dznCt8xPW7o&t=860s) pretty well. Uh, you know, we got a little bit lucky with the result, but

[14:21](https://www.youtube.com/watch?v=dznCt8xPW7o&t=861s) little bit lucky with the result, but

[14:22](https://www.youtube.com/watch?v=dznCt8xPW7o&t=862s) little bit lucky with the result, but was really missing is these like

[14:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=863s) was really missing is these like

[14:23](https://www.youtube.com/watch?v=dznCt8xPW7o&t=863s) was really missing is these like accounting checks or this like kind of

[14:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=864s) accounting checks or this like kind of

[14:24](https://www.youtube.com/watch?v=dznCt8xPW7o&t=864s) accounting checks or this like kind of human in the loop uh uh activity. So, uh

[14:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=867s) human in the loop uh uh activity. So, uh

[14:27](https://www.youtube.com/watch?v=dznCt8xPW7o&t=867s) human in the loop uh uh activity. So, uh and a lot of these accounting checks

[14:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=868s) and a lot of these accounting checks

[14:28](https://www.youtube.com/watch?v=dznCt8xPW7o&t=868s) and a lot of these accounting checks that my dad came up with, you know,

[14:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=869s) that my dad came up with, you know,

[14:29](https://www.youtube.com/watch?v=dznCt8xPW7o&t=869s) that my dad came up with, you know, we're were able to to codify. So, one

[14:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=872s) we're were able to to codify. So, one

[14:32](https://www.youtube.com/watch?v=dznCt8xPW7o&t=872s) we're were able to to codify. So, one was the the executive 45% like my dad

[14:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=874s) was the the executive 45% like my dad

[14:34](https://www.youtube.com/watch?v=dznCt8xPW7o&t=874s) was the the executive 45% like my dad said. Um so, it's really important to

[14:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=876s) said. Um so, it's really important to

[14:36](https://www.youtube.com/watch?v=dznCt8xPW7o&t=876s) said. Um so, it's really important to have these like domain specific checks

[14:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=878s) have these like domain specific checks

[14:38](https://www.youtube.com/watch?v=dznCt8xPW7o&t=878s) have these like domain specific checks when you're building uh these these AI

[14:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=880s) when you're building uh these these AI

[14:40](https://www.youtube.com/watch?v=dznCt8xPW7o&t=880s) when you're building uh these these AI agents. You know, I don't come from the

[14:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=881s) agents. You know, I don't come from the

[14:41](https://www.youtube.com/watch?v=dznCt8xPW7o&t=881s) agents. You know, I don't come from the from the accounting world. when I built

[14:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=882s) from the accounting world. when I built

[14:42](https://www.youtube.com/watch?v=dznCt8xPW7o&t=882s) from the accounting world. when I built this and I saw 4% I was like oh amazing.

[14:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=885s) this and I saw 4% I was like oh amazing.

[14:45](https://www.youtube.com/watch?v=dznCt8xPW7o&t=885s) this and I saw 4% I was like oh amazing. Um but but really it's um it's yeah this

[14:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=888s) Um but but really it's um it's yeah this

[14:48](https://www.youtube.com/watch?v=dznCt8xPW7o&t=888s) Um but but really it's um it's yeah this is really the number one thing to

[14:49](https://www.youtube.com/watch?v=dznCt8xPW7o&t=889s) is really the number one thing to

[14:50](https://www.youtube.com/watch?v=dznCt8xPW7o&t=890s) is really the number one thing to remember from this talk. If you take one

[14:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=891s) remember from this talk. If you take one

[14:51](https://www.youtube.com/watch?v=dznCt8xPW7o&t=891s) remember from this talk. If you take one thing away um having a human in the loop

[14:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=893s) thing away um having a human in the loop

[14:53](https://www.youtube.com/watch?v=dznCt8xPW7o&t=893s) thing away um having a human in the loop or ideally a subject matter expert uh in

[14:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=895s) or ideally a subject matter expert uh in

[14:55](https://www.youtube.com/watch?v=dznCt8xPW7o&t=895s) or ideally a subject matter expert uh in the loop is really really really

[14:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=897s) the loop is really really really

[14:57](https://www.youtube.com/watch?v=dznCt8xPW7o&t=897s) the loop is really really really important. Um they can take a look at

[14:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=899s) important. Um they can take a look at

[14:59](https://www.youtube.com/watch?v=dznCt8xPW7o&t=899s) important. Um they can take a look at your agent or your model or or whatever

[15:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=901s) your agent or your model or or whatever

[15:01](https://www.youtube.com/watch?v=dznCt8xPW7o&t=901s) your agent or your model or or whatever you've actually built. They can check

[15:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=902s) you've actually built. They can check

[15:02](https://www.youtube.com/watch?v=dznCt8xPW7o&t=902s) you've actually built. They can check the assumptions, check the inputs, check

[15:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=904s) the assumptions, check the inputs, check

[15:04](https://www.youtube.com/watch?v=dznCt8xPW7o&t=904s) the assumptions, check the inputs, check the outputs, make the right corrections.

[15:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=906s) the outputs, make the right corrections.

[15:06](https://www.youtube.com/watch?v=dznCt8xPW7o&t=906s) the outputs, make the right corrections. you can codify their checks in the model

[15:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=908s) you can codify their checks in the model

[15:08](https://www.youtube.com/watch?v=dznCt8xPW7o&t=908s) you can codify their checks in the model and it it just makes it into something

[15:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=910s) and it it just makes it into something

[15:10](https://www.youtube.com/watch?v=dznCt8xPW7o&t=910s) and it it just makes it into something that's a lot more autonomous and uh uh

[15:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=913s) that's a lot more autonomous and uh uh

[15:13](https://www.youtube.com/watch?v=dznCt8xPW7o&t=913s) that's a lot more autonomous and uh uh uh just a lot a lot better overall. And

[15:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=916s) uh just a lot a lot better overall. And

[15:16](https://www.youtube.com/watch?v=dznCt8xPW7o&t=916s) uh just a lot a lot better overall. And that is it. Thank you all so much for

[15:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=918s) that is it. Thank you all so much for

[15:18](https://www.youtube.com/watch?v=dznCt8xPW7o&t=918s) that is it. Thank you all so much for coming. Thank you.