Right Answer, Wrong Reasons: AI Forecasts Costco - Hassan El Mghari & Omar El Mghari, Together AI

Could an open language model take Costco’s public filings and build a forecast of its profits over the next eight quarters? Hassan El Mghari, who led developer experience at Together AI, built the app with his father, Omar El Mghari, a former CFO and head of internal audit. When Hassan saw how closely the forecasts tracked Costco’s results, he thought the experiment had worked. Omar wanted to see the assumptions.

Membership pricing gave them a change worth modeling. Costco sold roughly $300 billion of merchandise a year at close to cost. Hassan put operating profit at about $12 billion and attributed a little over half of it to membership fees. In July 2024, Costco announced its first fee increase since 2017. Standard membership would go from $60 to $65 and executive membership from $120 to $130. Each executive renewal at the new price would therefore bring in twice as much additional cash as a standard renewal.

The team set the information cutoff a couple of weeks before the announcement. The question was conditional. Given what Costco had disclosed by then, how would revenue and profit change if the company raised its fees? The team could supply the pricing decision, calculate its consequences and compare the forecast with later results.

Restricting the filings solved only part of the problem. Asked whether Costco had changed its membership fees in 2024, the first open models the team tried could identify the increase. Older candidates missed it. The team chose GPT-OSS-120B to build the financial models. The screening question could reveal prior knowledge, though a missed answer alone could establish little about everything a model had seen in training.

The working material came from about 214 public filings, including quarterly and annual earnings reports and sales reports. The team turned those documents into roughly 3,000 timestamped facts, then filtered on when each fact was available. Sixteen parallel sessions received the pre-cutoff material. Each session had to turn those facts into formulas the team could inspect and run.

Suppose a member pays $65 today. How much fee revenue belongs in this quarter? Costco’s quarterly report said the company recognizes membership fee revenue on a deferred basis. GPT-OSS-120B translated that policy into structured JSON specifying ratable recognition over a term of 52 weeks. Those two choices tell the calculation how to divide the payment and over what period.

At $65 divided by 52, each week of membership contributes $1.25 of revenue. The member pays once, while each quarter picks up only the portion earned during that quarter. Writing the rule as a formula let the language model interpret the filing once and left the arithmetic to the financial model.

Hassan laid the financial model out in Excel. It combined 14 generated formulas, figures taken directly from Costco’s filings and assumptions supplied by the language model. Two of those assumptions would matter especially. How many members were in the executive tier, where the increase was $10? How many were in the US and Canada, where the increase applied?

A formula that depended on itself failed a structural check. A build in which a higher fee failed to raise revenue failed a behavioral check for this scenario. When a check failed, the system sent the error back to the session that wrote the formulas and gave it up to two chances to repair the build.

Five builds passed and went into the forecast comparison. The other eleven still failed when their repair attempts ran out. The five thus qualified on the behavior of their formulas, before the team compared their forecasts with Costco’s reported results.

Hassan had started by tuning a single prompt, trying to get one build to pass all the checks. That took enough engineering time to make parallel attempts attractive. The sixteen runs together cost 19 cents. At that price he could afford to discard eleven candidates. What the other five had in common depended on what he had thought to test.

In the app, Hassan asked what would happen if Costco raised its fee that September. One of the newer language models had a narrower job here. It translated the question into structured scenario data and passed those inputs to the financial model GPT-OSS-120B had already written. Evaluating the formulas took about 10 milliseconds after translation. The app returned a projected $560 million increase in profit over two years.

How closely did predicted profit track the results Costco later reported? Hassan plotted the prediction in blue and the earnings-report figures in orange. He reported an average error of about 4.4% across the five passing builds, with individual errors ranging from 0.7% to 5.2%. The error formula was unspecified, which limits what that comparison tells us about the separate $560 million estimate of incremental profit. Hassan found the agreement encouraging enough to show his dad.

Omar started by looking at which assumptions could move the forecast. Changing the executive share produced large swings. Changing the US-and-Canada assumption produced essentially no movement in sales or fees. That second result troubled him. If only members in those countries faced the increase, changing the number of members there should change the forecast.

Where did the executive share of 15% come from? Its citation led to a genuine statement that executive membership penetration had increased. That established a direction of change. The language model supplied the percentage itself, along with a range of 5% to 30%. Costco’s filings already disclosed 45%. An analyst could have tested every value in the proposed range and still missed the figure in the filing.

The audit traced the close result to two inputs that pushed the fee estimate in opposite directions.

• Too many affected members. The formulas applied the increase to Costco’s worldwide membership. Members outside the US and Canada contributed a price increase they would never pay. That explained why changing the geographic assumption had done so little.
• Too few executive members. At a 15% executive share, too few members paid the $10 increment and too many paid the $5 increment. The resulting average increase per member was too small, partly concealing the inflated member count.

Omar also compared estimates of the fee increase itself. Across the sixteen attempts, those estimates ranged from about $500 million to $6 billion. His recalculation put the increase at $450 million. Even the engineering check that required revenue to rise with price allowed plenty of room for a wrong answer.

What happens if you fix just the executive share? In Omar’s diagnostic example, the error started at about 1.7%. He replaced 15% with the disclosed 45% and left the worldwide member count in place. The error jumped to 28%. The formulas now assigned the larger increase to more members of a population that was already too large. Putting one assumption right made the remaining mistake easier to see.

Omar borrowed three lines of defense from his auditing work to explain who could catch which failure.

• Management controls. Hassan’s engineering checks could catch a self-reference or a fee increase that moved revenue in the wrong direction.
• Internal accounting review. An auditor could trace a sensitive assumption back to the number in a filing, or change the affected member count and see whether the forecast responded. Both tests mattered here.
• Independent review. Omar proposed external auditors as a further check on the work of the people who built and reviewed the model.

The expert’s work could carry forward into the next build. Hassan and Omar turned several accounting checks into code, including one for the disclosed 45% executive share in this case. A future candidate could face that check automatically. Omar still saw a job for professional judgment. Someone familiar with the business had to notice that an input as important as the number of affected members could change without moving the forecast, then decide what the software should test.