# Your Company Has No Model of Itself. Agents Can Build One - Joao Wenzel, Citadel Securities

Joao Wenzel, Citadel Securities

[Talk page](https://ai.engineer/talks/jDwc0aXTAFg-your-company-has-no-model-itself-agents) · [Video](https://www.youtube.com/watch?v=jDwc0aXTAFg)

## Your Company Has No Model of Itself. Agents Can Build One

João Wenzel explains how Citadel Securities builds a history of what teams reported, what the evidence supported and what happened next, down to the unresolved dependency on an otherwise green slide.

### What the colleague who survived seventeen reorgs knows

[0:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=12s)

Think of the person at a company who has been through seventeen reorgs. They know the gossip and the politics, and somehow they can predict what will come of a big strategic move. João Wenzel calls that person the company’s world model. Colleagues can ask them how a decision is likely to play out, but there is only one of them, and hardly anyone writes down what they say. At Citadel Securities, Wenzel and his colleagues have been trying to approximate that capability with agents.

Citadel Securities builds its own technology down to the silicon, and more than two-thirds of its employees write code. Leadership had seen what coding agents could produce and wanted to know whether they could also help define and refine company strategy. Borrowing a visual from Tim Urban, Wenzel breaks that job into three questions. Where do we stand now? What happens next? How did we get here?

Each question requires some institutional memory. Two teams that used to collaborate may have stopped speaking after a reorg. A project’s current scope may make sense only to someone who remembers why it was narrowed several times. Even a meeting record takes interpretation. Who was in the room, what were they willing to say, and what did they leave unsaid?

### Two views of a project and three tests for new evidence

[2:19](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=139s)

The top-down view comes from strategy documents, decks, meeting reports and status updates. These are tidy accounts of what people want the organization to know. Their authors also exercise judgment about what to include and how to characterize it. Wenzel calls the extreme case a “watermelon project,” green on the outside and red on the inside. The bottom-up view comes from meeting notes, Slack threads, code changes and sessions with coding agents. A detail that explains a project’s condition may be buried in the middle of a much noisier exchange.

Wenzel borrows the state estimator from robotics to bring those views together. Start with a belief about something that matters, such as whether a project can meet its target. As time passes without a supporting observation, that belief grows less certain. A deck, a thread or a meeting note supplies another observation. It can give the system grounds to renew its confidence or to change its assessment.

Three considerations govern that update. How long has it been since the signal arrived? Where did it come from, and is that source still trustworthy? Does it agree with the other sensors? The third question matters even for recent evidence. A person may have posted a sincere assessment five minutes ago while another participant already has measurements that call it into question.

### Reading a status slide takes both XML and vision

[4:04](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=244s)

Consider the monthly status deck that someone in business operations already produces. Unpack the PowerPoint file and a script can read its XML. Now the script needs to decide what green is. Which hex code counts as the green that means a project is on track? A person can read the intended status at a glance, while the file exposes a color value whose significance the extractor has to establish.

Sending a screenshot of the slide to a vision model solves a different part of the problem. The model can interpret the visual presentation, but the slide’s author may have pasted in a screenshot to begin with. Even a high-resolution image of the finished slide can leave Claude struggling to distinguish 31 from 81 in the embedded image. Wenzel’s team uses both the PowerPoint XML and the vision model’s output to extract the evidence.

![Curved lines connect a highlighted table cell and the endpoint of a falling line chart to the same XML phrase, “31 (threshold 50).”](https://ai.engineer/api/talk-essay-media/a8a8d3f6f7991f341d4b6ea329adfb8a254ba2a14a634be196ad4c5296dc614b)

Read the picture and the XML together

Accurately reading one slide still leaves two kinds of change to account for. Across decks, a project spontaneously changes names in March, and the system needs to preserve its history. Between decks, the evidence can change in five days. Someone writes in Slack that the project looks on track. A colleague replies with a picture of a dashboard that undermines that assessment. Reading only the deck misses the exchange. Reading only the message misses the reply.

### What a March 16 status query misses

[6:11](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=371s)

The system organizes its work into five stages. Update carries the record forward. Age accounts for the time since a claim was supported. Observe examines incoming material for evidence. Grade compares the system’s expectations with what subsequently happened. Correct gives people a way to supply feedback and steer future judgments. Citadel Securities uses agents throughout this loop. The model proposes changes, and code and people decide how to update the records.

![Clockwise arrows connect Update, Age, Observe, and Grade. Each step names its worker or operation. A separate arrow labeled “Correct” shows where a person can intervene.](https://ai.engineer/api/talk-essay-media/9fbe8d86c475a80653bccf879b79e9f486e01285a7656d5e57cc6aaa5c3b2928)

Every step of the loop has its own workers

Wenzel illustrates the difference with a retrieval-augmented generation (RAG) query. Ask on March 16 whether a project is on track. A model can retrieve a couple of reassuring Slack messages and slides and confidently answer green. Those sources explain why someone might say green. To advise a manager, the agent also needs to know which project the sources describe, how its commitments have evolved and whether an outstanding issue qualifies the latest report.

### Carry Project North Star forward one month at a time

[7:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=449s)

For the monthly update, the inputs are last month’s project record and this month’s slide. Pairing them gives the agent a specific continuity question to resolve. When the name changes in March, the people involved and the goals they are pursuing still connect the new entry to Project North Star. The agent can infer that this is the same undertaking and carry its history forward under the new name. At the time of Wenzel’s presentation, the team had accumulated three years of these records.

The update distinguishes four things a single status field would collapse. It records what the team reported, the evidence the agent found, the assessment that evidence supports and the reason for the assessment. In Wenzel’s example, green is the reported status. A comment someone left on the slide identifies an outstanding dependency. The agent flags the project as yellow and cites the dependency as its reason. A manager can see both why the project appeared healthy in the deck and why it deserves another look.

![A JSON project record highlights two names under one ID and separate fields for reported status, evidence and the reason they differ. A loop diagram and three update rules appear to the right.](https://ai.engineer/api/talk-essay-media/a21ebee7d3e2280194771b537dffda282367ec5b191eebdb45ab481330aa5e63)

This is what the update step writes

Corrections flow into the next record. Earlier months stay as they were, including the judgments the agent could reasonably have made with the evidence then available. That choice gives the team an audit history across renames and revisions. It also preserves something a continuously rewritten summary would lose, namely the ability to test a February judgment using only what was knowable in February.

### Delivery dates show how quickly a claim ages

[9:11](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=551s)

In January, the projected go-live is Q2. It moves to Q3, then to October. Each target belongs in an account of how the project got here. To answer when it is likely to go live today, the system needs the date on which each target was claimed. Otherwise, a January slide promising Q2 remains available to support an answer long after the team has moved on.

The three-year history gave Wenzel a way to measure this effect. On average, half of delivery dates changed within two months. Of the dates that changed, 90 percent moved later than originally proposed. This gives practical meaning to his observation that a fact has a half-life. Even before a replacement target arrives, the age of the last estimate matters. Attaching a date to each claim lets an agent weigh recency when answering questions about a project, a person or the company’s intentions.

![Two 100-dot grids compare delivery dates and project stages. Half the delivery dates changed within 2 months, and half the project stages changed within 3 months. Between the grids, a time marker sits at 3 months.](https://ai.engineer/api/talk-essay-media/068bb638f7a1b805f8b67bbb99241d5ba27ea108dfe86246c4faf212d407677a)

Every fact has a half-life

### Specialist observers found more details worth recording

[10:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=612s)

The team’s first approach to observation was to give Claude or another model the entire collection of materials and see what it found. The appeal was simplicity. The failure was attention to detail. A model could take in the corpus and still overlook individual observations that should change a project record, much as a reader scanning the green indicators could miss the comment about a dependency.

The team broke the job into narrower assignments. Specialist agents examine the materials for particular kinds of information, and other agents review their findings. Wenzel reports that the number of updates the team considered insightful for a project more than doubled with this workflow. That comparison concerns what the observers found worth recording. The team used a separate evaluation to learn whether any of those observations helped identify future trouble.

![Five specialists each take one slice of work. Crossing lines connect them to reviewers, whose outputs converge at a code step that counts and retries. An arrow leads to a 10-by-10 grid with the lower cells highlighted in teal.](https://ai.engineer/api/talk-essay-media/672a05362f0297a447f101cb19efc8bc1e0f3a520ddda252484d0bea0a168cac)

Break the work up, and count what comes back

### Back-test a February warning against the next status label

[11:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=672s)

To grade the model, pick a date in the past and reconstruct the record as it stood then. Wenzel uses February as the example. Give the agent the evidence available by that point and see which apparently healthy projects it flags as likely to slip. Then consult the status labels that arrived afterward. Because later updates were carried forward, a warning discovered in March cannot silently improve the agent’s February call.

Wenzel reports that projects flagged while green were twice as likely to turn yellow or red in later status reports. He leaves the comparison group implicit. A manager can use the flag to decide which project to inspect first, even while the cause of a future status change and its effect on delivery remain open questions.

![Two grids compare projects that started green. Yellow squares mark the projects that turned yellow or red within three months: 18 out of 100 flagged projects and 10 out of 100 unflagged projects.](https://ai.engineer/api/talk-essay-media/d6922d1becb1d895d4a5a63ae64268605236ba82ee48a98a9fa99230c80af8a7)

Flagged greens were about twice as likely to turn yellow or red

### Ask stakeholders for the rule behind a correction

[12:07](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=727s)

Wenzel describes having 18 cases with a flagged verdict, then turns to the cases where the agent lacks a verdict. This is partly a product-design problem. Someone with the missing context has to see the question and find it easy enough to answer. The team put work into collecting feedback in the app a stakeholder already uses, in Slack or by email. Every additional step between seeing the question and answering it makes that loop harder to sustain.

The wording determines how far a correction travels. Asking for a fact can settle one disputed output. Asking about the rule the person used to reach that judgment can improve other assessments to which the same rule applies, now and in future updates. The project record distinguishes what the agent initially inferred from what a person subsequently confirmed. A later reader can tell where human confirmation entered the history.

![Two questions about project status receive the same answer, “yes.” A single outlined cell shows the scope of the fact, while rows of turquoise cells show the scope of the rule.](https://ai.engineer/api/talk-essay-media/a06b08c9cf20d0003e2b94d36606c9a24c292c2834e971f6a09c3f36cb9754fe)

Ask people about rules, not facts

### Read the thread, the pull request and the work that preceded it

[13:13](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=793s)

Bottom-up sources expose different parts of the work. Meetings capture what people say aloud. Slack offers a continuous, noisy account close to the work itself. The unit the team examines is a thread, including its participants and attachments. For a claim worth recording, the agent identifies when it was made, who was responsible or whom it concerns, which project it belongs to and where to find the supporting evidence. That last link makes it possible to revisit the dashboard image in its original exchange.

![A thread combines an on-track update, a test dashboard with rising red bars, and a report of failed imports. Arrows lead through screenshot captioning and thread analysis to a dated Atlas Ledger Migration claim marked “At risk.”](https://ai.engineer/api/talk-essay-media/1d166649697e97354f72f6da4ab6d3151928c9a437800d8fb1c35fa059c13bf0)

Distill each thread into dated claims

Pull requests show what changed and how a project is evolving. For engineers who opt in, coding-agent session traces also show how the work reached the pull request. The original ask may have evolved, tests may have failed along the way, or the developer may have made a decision that explains the final change. Reading the trace alongside the PR gives the system access to decisions and difficulties that the finished change alone may leave implicit.

Collection itself can fail. The system once treated a channel that appeared to be going quiet as evidence about the people or project using it. Wenzel compares that inference to driving down a highway and concluding that your speed has dropped to zero because the speedometer reading disappeared. The team began tracking two times for each source, when it last tried to collect information and when collection last succeeded. If those times diverge, the apparent quiet may tell the manager more about the collector than about the project.

### Open the leadership meeting where the evidence disagrees

[15:35](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=935s)

Return to the green project. The assembled record shows that the go-live has slipped twice and people are discussing failing tests. The evidence points to yellow. Leadership can open with that discrepancy and discuss the work behind it, sparing everyone another pass through slides they have already skimmed. The records also feed a weekly read for managers and leaders on what stuck and what went quiet.

Wenzel sees a further use in giving other agents a common source of context. In its client business, Citadel Securities is exploring whether a similar model could bring internal and external evidence together to explain who holds what, what clients are interested in and how to anticipate their needs. That extension exposes a harder identity question than the North Star rename. One client can have four records in four systems with no canonical identifier. Wenzel says the team still lacks a clear way to establish that those records belong together.

Stakeholder attention is another unresolved constraint. Wenzel distinguishes three cases. For a low-stakes update the model is confident about, apply it automatically. For a low-stakes update it is unsure about, wait for more evidence. When the stakes are high, the choice of whom to ask, when to ask and how to collect the answer matters. He closes by encouraging teams to start collecting history early. Thousands of decks gave his team material to work with, but the dated observations, subsequent outcomes and human corrections are what made that history usable in daily work.

## Key takeaways

- Keep the reported status, supporting evidence, agent assessment and rationale separately. Carry corrections forward so an earlier judgment can be evaluated on the evidence available at the time.
- Date each claim and track both the last collection attempt and the last successful read. An old estimate and a silent collector call for different interpretations.
- Evaluate observations and predictions separately. Count the useful updates a workflow finds, then back-test its flags against later outcomes with a clearly defined comparison group.
- When a stakeholder can explain a recurring judgment, capture the rule and distinguish their confirmation from the agent’s original inference.

## Slides

### Agents help with the grind, and people make the decisions

[1:08](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=68s) [A three-column table pairs five stages of an investment professional’s work with human decisions and the tasks agents help prepare.](https://assets.ai.engineer/sites/aiecode2025/sha256/baea38768a6d25905aae1b3fc3050b5beb6dc99837cab8bdd62badcacb5b959c.webp)

Agents help with the grind, and
people make the decisions

Investment professional	The human edge	What agents help prepare
Absorb information	Spot the five percent insight	Catalog and structure information
Originate a trade	Surface information relevant to trade ideas	Monitor trends, alert on events
Pressure-test it	Sign off on the trade	Draft risk analysis and summaries for review
Implement	Influence partners	Suggest instruments
Monitor		Monitor performance

AI at Citadel Securities

### Strategy comes down to three questions

[1:30](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=90s) [A highlighted path runs from the company’s founding to its present position. Gray paths show other ways the past could have unfolded, and teal paths fan out into possible futures.](https://assets.ai.engineer/sites/aiecode2025/sha256/25a40024c8a1932ef9ea6b5bd95799a41355d26a76db5c4ad2e8f968963e35a9.webp)

Strategy comes down to three questions

Founded
How did we get here?
Where do we stand?
What happens next?

The past	The future
After Tim Urban, Wait But Why

A model of the company

### The hard part is knowing how it got here

[1:43](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=103s) [A timeline from years ago to today marks two teams losing contact after a reorganization, a project changing scope twice, and the people whose agreement a decision requires.](https://assets.ai.engineer/sites/aiecode2025/sha256/b873ecbd5c74344aafbeed18b311409da082fd582813c8928fa5d173bd74ff57.webp)

The hard part is knowing how it got here

years ago

Relationships
Two teams that
stopped talking
after a reorg

A project’s story
Why it was rescoped twice

The politics
Who really decides,
and who has to agree

Today

What someone who’s been here ten years just knows

A model of the company

### There are two ways in, top down and bottom up

[2:17](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=137s) [Two arrows converge on a company model. One points down from decks and reports. The other points up from meetings, Slack, code changes and agent sessions.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fc646dc9b557e400c02869192e09bc8caf68abfca90e74e1f9b5d84605d5796.webp)

There are two ways in, top down and bottom up

Top down
Decks, strategy docs, status reports
somebody’s summary, late, and sometimes green outside, red inside

The model is where they meet

Bottom up
Meetings, Slack, code changes, agent sessions
close to the truth, noisy, and years of it

A model of the company

### The belief updates, and the uncertainty shrinks

[3:36](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=216s) [A tight cluster of dots sits inside a much larger dashed circle under “Project status.” “Update” is highlighted beside the diagram, with the belief update equation below.](https://assets.ai.engineer/sites/aiecode2025/sha256/71cbcd409b6da1766999e928f95c1b35a3ef80262db73529a5e2078135d4081d.webp)

The belief updates, and the uncertainty shrinks

Project status
Weeks with nothing new

Age
Observe
Update

belief[t] = update(belief[t-1], evidence[t])

A state estimator

### Start at the top, with the deck leadership reads

[4:05](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=245s) [A curved line connects an agent’s question about projects to the monthly status deck. An inset shows a project table with green and yellow status cells and a downward-sloping line chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d501c68ddd221e17e8b0f4846eeefb38ee1a43020144c3fc6d68ec7d77d0782.webp)

Start at the top, with the deck leadership reads

“How are our
projects going?”

The monthly status deck
the summary leadership reads,
written by the people being measured

Platform Services · Project status
Monthly operating review · as of 26 Feb 2026
Area	Project	Owner	Status	Go-live	Commentary
Core Platform
Controls

Top down: the slide

### Green is three different colors

[4:30](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=270s) [Three cells in a project status table all say “On track.” Lines connect each cell to a different green swatch and its hex code.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa8b971cf30ba201ceb20eed147786a5b7171e5048a4fcfaff57ff3e1ca4b429.webp)

Green is three different colors

Platform Services · Project status
Monthly operating review · data as of 6 Feb 2026
Platform Services  PS

Area | Project | Owner | Status | Go-live | Commentary
Core Platforms | Northstar | Dana Hollis | On track | Q3 2026 (was Q2 2026) | Failing tests down to 31 (threshold 50). Parallel run starts 10 Feb; cutover dry-run Fri 13 Feb. Upstream clearing API v2 still outstanding (missed the clearing team's February release); UAT start moved out four weeks. Go-live re-planned to Q3.
Cash Forecasting Rebuild | Dana Hollis | On track | Jun 2026 | Intraday cash feeds connected for 3 of 5 banks; first forecast backtest within 2% of actuals.
Vendor Onboarding Portal | Soren Lindqvist | On track | Apr 2026 | Pilot expanded to 30 suppliers; bulk upload and status emails added from pilot feedback.
Controls | Access Review Automation | Talia Vantreight | At risk | Q3 2026 | Vendor due diligence running three weeks late; security questionnaire outstanding. Shortlist of two holds; Q3 at risk if diligence does not close by mid-March.

Status: G on track · A at risk · R off track
Mock conversion runs - failing tests - last 6 weeks
Source: PMO tracker · RAG as reported by project owners · internal

Parse the file
Show a model the picture
92D050
00B050
70AD47

Top down: the slide

### A vision model can barely read a picture this small

[4:45](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=285s) [A blurred project report appears beside a thumbnail labeled “What the model gets.” A line from the report’s pixelated chart points to three possible readings of a number. Both reading methods are crossed out.](https://assets.ai.engineer/sites/aiecode2025/sha256/b83ac158e4d3a76e1a9727205fa52ba5861bdf18ac8f4f8c92ff900eae1caea9.webp)

A vision model can barely read a picture this small

Platform Services · Project status
Area | Project | Owner | Status | Go-live | Commentary
Core Platforms | Northstar
Cash Forecasting Rebuild
Vendor Onboarding Portal
Controls | Access Review Automation

Parse the file
Show a model the picture

What the model gets
31?  81?  310?

Top down: the slide

### Read the picture and the XML together

[4:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=299s) [Curved lines connect a highlighted table cell and the endpoint of a falling line chart to the same XML phrase, “31 (threshold 50).”](https://assets.ai.engineer/sites/aiecode2025/sha256/a8a8d3f6f7991f341d4b6ea329adfb8a254ba2a14a634be196ad4c5296dc614b.webp)

Read the picture and the XML together

Platform Services · Project status
Monthly operating review · data as of 6 Feb 2026
Platform Services  PS
Area | Project | Owner | Status | Go-live | Commentary
Core Platforms | Northstar | Dana Hollis | On track | Q3 2026 (was Q2 2026) | Failing tests down to 31 (threshold 50).
Cash Forecasting Rebuild | Dana Hollis | On track | Jun 2026
Vendor Onboarding Portal | Soren Lindqvist | On track | Apr 2026
Controls | Access Review Automation | Talia Vantreight | At risk | Q3 2026
Status: G on track · A at risk · R off track
Mock conversion runs - failing tests - last 6 weeks
2 Jan   9 Jan   16 Jan   23 Jan   30 Jan   6 Feb

<a:tc>
  <a:txBody><a:p><a:r>
    <a:t>Failing tests
      down to
      31 (threshold 50).
      Parallel run…</a:t>
  </a:r></a:p></a:txBody>
</a:tc>

The picture: color, layout, charts
The XML: the exact words

Top down: the slide

### Now a new deck arrives every month

[5:13](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=313s) [Five project reports sit above a January–May timeline. A callout highlights the March row where Northstar becomes Atlas Ledger Migration. The timeline continues with a dotted line.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ba210411e2b98bebe5dfcbed3f83c5b7e4a00986ae99cfbe432447e8c0b7037.webp)

Now a new deck arrives every month

Northstar becomes Atlas Ledger Migration

January     February     March     April     May     June, July…

Top down: the slide

### Your company has no model of itself

[5:41](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=341s) [Dotted lines lead from a green “On track” status to a message about Friday’s cutover, a screenshot with rising red bars, and a dictated meeting note.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8d99e80303f2d388cdc16b7ec24c82246b6945f16dc5665fd16b14f682268af.webp)

Your company has no model of itself

On track
The deck says green

“Cutover dry-run is on track for Friday”
Three days later, failing tests are climbing

A screenshot in a reply
Ledger migration · Parallel run test results
212     97.1%     7,318     1 of 4

A note dictated after a client meeting

Top down: the slide

### Every step of the loop has its own workers

[6:41](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=401s) [Clockwise arrows connect Update, Age, Observe, and Grade. Each step names its worker or operation. A separate arrow labeled “Correct” shows where a person can intervene.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fbe8d86c475a80653bccf879b79e9f486e01285a7656d5e57cc6aaa5c3b2928.webp)

Every step of the loop has its own workers

Update
a reconciler, then an updater
code applies

Age
a cleanup pass

Observe
readers, one per source

Grade
a backtest

Correct
a person

The model proposes    Code and people decide

The loop

### Carry one record per project forward, a month at a time

[7:19](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=439s) [An arrow carries Northstar’s January record into the model alongside February’s slides. Both records show green. The go-live moves from Q2 to Q3 2026, and February adds a note about the clearing feed.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf2ffa9d28b1582db6c42e55783e92b09762c0b0d6f0bc7520e27807a39784ff.webp)

Carry one record per project
forward, a month at a time

January
slides
model
Northstar
On track
Go-live Q2 2026

February
Northstar
On track
Go-live Q3 2026
Waiting on the clearing feed

March, and every month after

Last month’s record + this month’s slides → this month’s record

Update · Age · Observe · Grade · Correct

### Let the model pick from the projects you already know

[7:53](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=473s) [An arrow connects March’s Atlas Ledger Migration row to Northstar in the list of known projects. The resulting record retains Northstar as an alias and shows a green status and an October 2026 go-live.](https://assets.ai.engineer/sites/aiecode2025/sha256/65d79dda8c61f6ad8d927e0e6d248324aa5e5ce2b6f97b31ebe615aa2a2851b8.webp)

Let the model pick from the
projects you already know

Known projects
Northstar
Cash Forecasting Rebuild
Partner Portal
Access Reviews
New project

March’s row
Atlas Ledger Migration
same project, new name

Atlas Ledger Migration
also known as Northstar
On track
Go-live October 2026

Update · Age · Observe · Grade · Correct

### Keep what was claimed apart from what the evidence says

[8:22](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=502s) [A bracket and flag mark the discrepancy in Northstar’s 16 February record. The reported status is green, while the evidence is yellow because the clearing feed is still outstanding.](https://assets.ai.engineer/sites/aiecode2025/sha256/240bd758364ce8345e1a5a6d353e646ec7a7ced40d0ea3b685999e771e2d7012.webp)

Keep what was claimed apart
from what the evidence says

Northstar
as of 16 February
Reported: green
Evidence: yellow, a dependency is still outstanding
Go-live Q3 2026
Waiting on the clearing feed

About one in six green statuses got flagged

Update · Age · Observe · Grade · Correct

### Heal forward, never rewrite the past

[8:45](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=525s) [A monthly grid compares reported status with evidence. January has two green squares. February and March each have green reported status and yellow evidence. Locks mark January and February, an outline highlights March, and dotted squares mark April.](https://assets.ai.engineer/sites/aiecode2025/sha256/22d0f2f57082e9e8f40c9832fe88a0163f9d8002cc54d16307536db4afe6c808.webp)

Heal forward, never rewrite the past

Jan      Feb      Mar      Apr
Reported
Evidence

locked as believed then
now, it heals as evidence arrives

So you can ask what it knew on any date

Update · Age · Observe · Grade · Correct

### This is what the update step writes

[9:01](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=541s) [A JSON project record highlights two names under one ID and separate fields for reported status, evidence and the reason they differ. A loop diagram and three update rules appear to the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/a21ebee7d3e2280194771b537dffda282367ec5b191eebdb45ab481330aa5e63.webp)

This is what the update step writes

{
  "id": "proj-2210",
  "names": ["Northstar", "Atlas Ledger Migration"],
  "status": {
    "reported": "green",
    "evidence": "yellow",
    "reason": "clearing feed dependency outstanding"
  },
  "go_live": "2026-10"
}

The model proposes,
code applies

Identity is an ID,
never a name

Never rewrite the past

Update · Age · Observe · Grade · Correct

### Every fact has a half-life

[9:32](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=572s) [Two 100-dot grids compare delivery dates and project stages. Half the delivery dates changed within 2 months, and half the project stages changed within 3 months. Between the grids, a time marker sits at 3 months.](https://assets.ai.engineer/sites/aiecode2025/sha256/068bb638f7a1b805f8b67bbb99241d5ba27ea108dfe86246c4faf212d407677a.webp)

Every fact has a half-life

100 delivery dates
Half the delivery dates
changed within 2 months

0   1   2   3
months since
the deck said it

100 project stages
Half the project stages
changed within 3 months

Update · Age · Observe · Grade · Correct

### Every claim carries its age

[9:50](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=590s) [A project record highlights the date attached to its status and go-live claims. The status is reported green with yellow evidence. The October 2026 go-live date has moved twice. Both claims are dated March 10, 2026.](https://assets.ai.engineer/sites/aiecode2025/sha256/b8dcc5ef3f444d8dfd7c31d75326304b2015177a391d24bf19874c3abfa777be.webp)

Every claim carries its age

{
  "id": "proj-2210",
  "names": ["Northstar", "Atlas Ledger Migration"],
  "status": {
    "reported": "green", "evidence": "yellow",
    "reason": "clearing feed dependency outstanding",
    "as_of": "2026-03-10"
  },
  "go_live": {
    "value": "2026-10",
    "as_of": "2026-03-10",
    "moved": 2
  }
}

Every claim carries a date

Hand the agent the age,
not just the claim

Update · Age · Observe · Grade · Correct

### Break the work up, and count what comes back

[10:11](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=611s) [Five specialists each take one slice of work. Crossing lines connect them to reviewers, whose outputs converge at a code step that counts and retries. An arrow leads to a 10-by-10 grid with the lower cells highlighted in teal.](https://assets.ai.engineer/sites/aiecode2025/sha256/672a05362f0297a447f101cb19efc8bc1e0f3a520ddda252484d0bea0a168cac.webp)

Break the work up, and count what comes back

Specialists, one slice each
Reviewers check each other

Code counts,
and retries

What came back

Update · Age · Observe · Grade · Correct

### A world model should tell you what happens next

[11:10](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=670s) [A calendar, warning triangle, and building mark three questions about predicting project delays, changes from green to red, and client demand for a trade.](https://assets.ai.engineer/sites/aiecode2025/sha256/68b2a7b225a45a433a8723a77d3db0c43fc5ade3f085e3f38c1b154673c6e25c.webp)

A world model should tell you what happens next

Which project is about to slip?

Which green turns red next month?

Which client will want this trade next?

Update · Age · Observe · Grade · Correct

### The future is the label

[11:23](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=683s) [Rows of colored dashes run across a timeline from 2024 to 2026. Teal dashes mark what the model flagged in February 2026. A vertical line separates those flags from the subsequent marks labeled “Then.”](https://assets.ai.engineer/sites/aiecode2025/sha256/acc4b1ba6e9a9941f7c1b322c87ebcf1491b234d29d1941508834595397930a2.webp)

The future is the label

What it flagged in Feb 2026     Then

2024                     2025                     2026

Update · Age · Observe · Grade · Correct

### Flagged greens were about twice as likely to turn yellow or red

[11:43](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=703s) [Two grids compare projects that started green. Yellow squares mark the projects that turned yellow or red within three months: 18 out of 100 flagged projects and 10 out of 100 unflagged projects.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6922d1becb1d895d4a5a63ae64268605236ba82ee48a98a9fa99230c80af8a7.webp)

Flagged greens were about twice
as likely to turn yellow or red

Flagged greens
18

Unflagged greens
10

Turned within three months, out of 100

Update · Age · Observe · Grade · Correct

### Send people a question, where they already work

[12:04](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=724s) [A question about the clearing feed branches into three ways to answer it: in the app, in a Slack thread, or from an email inbox. Grade is highlighted in the footer.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe8329e8f46aadf9803c5469dfa6e5f8e6d7e4910bb88c1a97f2fab19bc8d465.webp)

Send people a question, where they already work

“The October go-live now depends on the
clearing feed, so when will it be ready?”

The app they’re using
answered right there

Slack
a reply in the thread

Email
one click from the inbox

As close to the work, and as easy, as you can make it

Update · Age · Observe · Grade · Correct

### Ask people about rules, not facts

[12:45](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=765s) [Two questions about project status receive the same answer, “yes.” A single outlined cell shows the scope of the fact, while rows of turquoise cells show the scope of the rule.](https://assets.ai.engineer/sites/aiecode2025/sha256/a06b08c9cf20d0003e2b94d36606c9a24c292c2834e971f6a09c3f36cb9754fe.webp)

Ask people about rules, not facts

A fact
“Is this project really yellow this month?”
yes
fixes one output

A rule
“Does an outstanding
dependency mean it isn’t green?”
yes
fixes every output it covers

Update · Age · Observe · Grade · Correct

### A person’s answer goes into the record too

[13:04](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=784s) [A project record shows a reported green status and yellow evidence. Three highlighted fields record confirmation, the employee ID, and the date. At right, a person icon points into a circular workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/93541117e9f31e69cdb4f2254b61a5b9d7f7e5caedc29bf9a3c92d62741a8c62.webp)

A person’s answer goes into the record too

{
  "id": "proj-2210",
  "names": ["Northstar", "Atlas Ledger Migration"],
  "status": {
    "reported": "green", "evidence": "yellow",
    "confidence": "confirmed",
    "confirmed_by": "employee:0417",
    "confirmed_on": "2026-03-18",
    ...
  },
  "go_live": { "value": "2026-10", "moved": 2, ... }
}

Ask about rules, not facts
Only people confirm
Key state on IDs

Update · Age · Observe · Grade · Correct

### Bottom up, you read the work as it happens

[13:16](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=796s) [Four rows pair work sources with what they reveal. Meetings and Slack capture what people say, code changes show what shipped, and the highlighted agent sessions show how it got made.](https://assets.ai.engineer/sites/aiecode2025/sha256/78b03a5a92f79938ea323ed4756cb402adb0c3e235da8f8bc84c4630e65752c5.webp)

Bottom up, you read the work as it happens

Meetings    the strongest signal, what people say out loud
Slack    constant, noisy, and close to the truth
Code changes    what actually shipped
Agent sessions    how it got made

Bottom up

### Distill each thread into dated claims

[13:35](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=815s) [A thread combines an on-track update, a test dashboard with rising red bars, and a report of failed imports. Arrows lead through screenshot captioning and thread analysis to a dated Atlas Ledger Migration claim marked “At risk.”](https://assets.ai.engineer/sites/aiecode2025/sha256/1d166649697e97354f72f6da4ab6d3151928c9a437800d8fb1c35fa059c13bf0.webp)

Distill each thread into dated claims

A thread
“Cutover dry-run is
on track for Friday”
212    1 of 4
“Data import still fails on
the old message format”

A vision model
captions
the screenshot

A model reads the
thread and pulls
out what matters

The claim it writes
12 February
The delivery lead
Atlas Ledger Migration
At risk
Failing tests climbing in the parallel run
A link back to the thread

Folded into the project’s record day by day, the way the decks fold month by month

Bottom up

### A PR shows what changed, and the session shows how it got there

[14:01](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=841s) [A merged pull request appears beside a six-step session timeline. The timeline highlights the person’s instruction to use the shared client after a test fails.](https://assets.ai.engineer/sites/aiecode2025/sha256/9338ec2add192a164d5e41cf6f66151fb13bec035820a8f1c58414caf4c98d90.webp)

A PR shows what changed, and
the session shows how it got there

Output
Pull request
Use the shared client
for clearing messages
Merged

Process
The ask: retry clearing messages that fail
The agent’s plan: write a new client
A test fails
The person: no, use the shared client
The agent tries again, and the tests pass
The pull request

The dead end and the correction never make it into the pull request

Bottom up

### Missing isn’t zero

[15:08](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=908s) [A chart spanning January to March shows seven solid bars of varying heights followed by four dotted placeholders for unreadable data. The Atlas Ledger Migration record lists the last successful read and the latest attempt.](https://assets.ai.engineer/sites/aiecode2025/sha256/2cd28280a45f79e917064e315a0bef76315eddd4a0169cfe0b3626f8cba6c450.webp)

Missing isn’t zero

couldn’t read, not zero
Jan   Feb   Mar

Atlas Ledger Migration
as of 16 Mar
Waiting on the clearing feed   unseen · last read 20 Feb · last tried 16 Mar

Bottom up

### Every source carries two dates

[15:13](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=913s) [The highlighted sources block shows matching attempt and success dates for the deck. Slack’s latest attempt was 16 March, its last successful read was 20 February, and its state is marked unseen.](https://assets.ai.engineer/sites/aiecode2025/sha256/86ce9783a2f0d587cfc799d286c6233839e2feb1a1b479d698c285db42c31434.webp)

Every source carries two dates

{
  "id": "proj-2210",
  "names": ["Northstar", "Atlas Ledger Migration"],
  "status": { "evidence": "yellow", ... },
  "go_live": { "value": "2026-10", "moved": 2, ... },
  "sources": {
    "deck": { "last_tried": "2026-03-10",
              "last_ok": "2026-03-10" },
    "slack": { "last_tried": "2026-03-16",
               "last_ok": "2026-02-20",
               "state": "unseen" }
  }
}

Count what comes back
Keep two dates per source
Unseen isn’t quiet

Bottom up

### Ask the record, and you get what’s true now

[15:25](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=925s) [Two boxes answer the same cutover question. Retrieval reports green for October. The record reports two slips, yellow evidence and unreadable Slack. Underlined fields in the project record show the supporting values.](https://assets.ai.engineer/sites/aiecode2025/sha256/78abc11860f490b03140c460258f7f1da64371cb8e72e8b43fdfc5a3144b3673.webp)

Ask the record, and you get what’s true now
“As of 16 March, is the Atlas ledger migration cutover on track?”

Retrieval
Green, on track
for October

Our record
No, the go-live slipped twice, the evidence says yellow, and the
project’s Slack hasn’t been readable since February

{
  "id": "proj-2210",
  "status": { "evidence": "yellow", ... },
  "go_live": { "value": "2026-10", "moved": 2, ... },
  "sources": {
    "slack": { "last_ok": "2026-02-20", "state": "unseen" }
  }
}

Where they meet

### What we actually use it for

[15:44](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=944s) [Four lines branch from the record to leadership conversations, a weekly update, meeting preparation and an agent context layer. The branch for the context layer under development is highlighted in teal.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf2bf9cd49b8f1881b918c42195816f3abb2fdb4b2dd0ee9f5ccd9d382bda77c.webp)

What we actually use it for

The record

Steer leadership conversations
start from where the deck and the evidence disagree, not from every slide

A weekly read on what’s stuck and what went quiet
sent to the people who can act on it

Meeting prep built from dated facts
every fact with a source and a date

A context layer every agent can rely on
we’re building it, so agents read the record with its dates and confidence

What it’s for

### Bring people in only where it’s worth their time

[16:56](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=1016s) [A two-by-two matrix crosses stakes with model confidence. Both high-stakes cells involve the owner. At low stakes, the system waits for evidence when unsure and applies the change automatically when sure.](https://assets.ai.engineer/sites/aiecode2025/sha256/dfcf012f2b9f060affe4aece6c50c4f2d4f5a3f3d93ade454b1acdef6b75c2ae.webp)

Bring people in only where it’s worth their time

High stakes	Ask the owner	Ask the owner to confirm, in one click
Low stakes	Wait for more evidence	Apply it automatically
	The model is unsure	The model is sure

And only a person’s yes ever writes confirmed

What’s still hard

### Your company can have a model of itself

[18:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=1092s) [A person icon points into a loop surrounding a record. Three numbered points describe estimating company state, testing it against later events, and assigning decisions to code and people.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bdd0c922f87cc5f0a96e939db7f1867d3aeea4750d3066eb7a772aa97a288a7.webp)

Your company can have a model of itself

1   Model your company as a state
    estimator, not a search index
    Retrieval tells you what was said, the record tells you what’s true now

2   Grade it against what happened next
    The future is the label, and every human answer is a test

3   Models propose, code and people decide
    Count in code, key on IDs, only people confirm

Start collecting now, and build your model from the exhaust

## Full transcript

[0:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=12s) Pleasure to be here. Uh, another round of applause for everyone at AI Engineering. This has been a really, really exceptional conference so far.  Thank you. All right, so I wanna begin by asking you to think about that person in your company who just seems to know everything. They've been through seventeen reorgs. They know all the gossip, all the politics, and somehow they always seem to predict what's gonna come out of these, like, big strategic moves. That person is a company's world model,

[0:42](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=42s) essentially, and the problem is that they don't scale, and nobody really writes down what they say.

[0:49](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=49s) I'm João Wenzel with Citadel Securities, and I'm gonna talk a little bit about how we try to approximate that with an agentic system.

[0:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=59s) So a little bit about Citadel Securities. We build our own technology down to the silicon. At this point, over two-thirds of our employees write code, so naturally, we are completely AI-pilled, and agents already help us with a lot of things at the organization. So naturally, leadership started asking questions about strategy. These harnesses are really good at producing high, you know, high-quality code, but can they actually help define and refine the strategy of our

[1:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=89s) company? So this is one of my favorite visuals I've borrowed from Tim Urban. Strategy really comes down to three questions, right? Where do we stand right now? What's gonna happen next? And how did we get here? And to answer any of those questions, an agent has to have a really rich profile of the company that it's operating in. So this is broadly a talk about, uh, engineering, uh, a institutional memory layer. Let's, let's, let's call it that.

[1:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=119s) The agents need to have information about relationships, right? Two teams that used to work together but no longer st- talk because there was a reorg, or a project story and why it was re-scoped several times, or the politics, right? Like, who's in the room? What are they saying, and what are they not saying? There's two ways in, right? There's top-down, and there's bottom-up. Top-down, you have the decks. You have your strategy docs, your meeting and status reports, and

[2:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=149s) they're tidy, and they're pretty clean. But there's a unique human twist. People generally like to infuse their thoughts and opinions into these things, sometimes by design, so you end up with what, you know, things I call watermelon projects, right? They're green on the outside and red on the inside. Then you have... On the bottom, you have the really noisy stuff, which has a surprising amount of signal, but it, uh, it's hard to uncover, right? You have meeting notes, and you have Slack and code changes and even your session traces with your favorite harness.

[2:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=179s) And the model is where they meet. The evidence is synthesized at that level. Now, the evidence piece sometimes is problematic because it's either partial or wrong. But it turns out that in robotics, you actually have the solution for this, and it's called a state estimator, right? What you do is you have some sort of belief about something you care about, and the longer it's been since you've seen or observed evidence that supports it, the more you-- the less you trust it, right? The fuzzier it

[3:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=209s) gets. Until you have a deck or if, uh, you know, some thread or some meeting note that sort of collapses that and you now have new evidence, right? And now you're more certain of it, and therefore, you produce the update. And the things that I care about when producing an update or that the agent should care about, there's three things. There's, how long has it been s- it been since some signal has been received? There's where did that signal come from, and is it still trustworthy? And then does it

[3:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=239s) broadly agree with other sensors, right? You can-- There's a lot of parallels here with, like, self-driving, for example, that I'll come back to. So let's start at the top, right? Like, a very simple example is something like a slide that is produced already by someone in biz ops, and they produce a monthly status deck. And the question is: How do you get data out of this slide, right? It's, uh... Many of you, I'm sure, have tried that. It's this-- It's deceptive actually, right? Because let's say you try to parse the file, right? Like, you unpack the PowerPoint file, and you get an

[4:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=269s) XML, and all of a sudden you get into this really interesting conceptual question. What is green, right? What is the hex code for green? Um, and you sort of-- Things break down when it comes to this type of script. Then the solution is to say, "Well, I'm just gonna take a screenshot of it and throw it at a vision model." But at that point, even if it's, like, pretty high res, you're dealing with someone who copy-pasted a screenshot back into a presentation and, like, essentially you have Claude trying to squint and trying to figure out if it's a thirty-one or an eighty-one. And

[4:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=299s) so, you know, the theme of the talk really is about, like, elegance comes from just synthesizing things and adding more information. And in this example, it's very basic, but you get the true picture by taking both the XML and the output of a vision model. Now, that's pretty basic, right? But the issue is that you want to establish some continuity over time, right? Like, these a- this collateral that your company accrues is very much sort of always building up. And so in this particular example, you have a pr- a

[5:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=329s) project that just spontaneously changed names back in March. And so, you know, you have to establish that sort of longitudinal traceable audit layer, uh, as you, as you create this shared context layer about the sort of institutional memory of your company. And the irony-- There's two things. The irony here is that even if you do this really well, you're still missing half the picture, right? You're looking at a deck, and your Slack says otherwise because that was five days ago, and now you have someone on Slack saying, "Hey, by

[5:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=359s) the way, I think we're on track for this," but then someone replies with a picture of a dashboard, and it's no longer the case. So things just start breaking down. And so- And we need to do a better job of, like, integrating all these things, right? Your company has no model of itself. So I'm proposing just a brief plan here, right? The idea is you take the three things that I described earlier about the state estimator. You have the update, age, and observe, observe, um, stages, and then you have grade. The grade

[6:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=389s) stage is basically comparing what you observed versus what you expected. And then the correct stage is basically a way for people to allow the system to get new feedback and steer the behavior of the system, right? At Citadel Securities, the system that we've built has agents at every part of this, uh, of this particular loop, right? But the key thing is that the model is proposing these changes, and then you have code and people deciding how the records are updated. And of course, you're probably asking yourself,

[6:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=419s) like, why not just RAG, right? Like, probably RAG would do the trick, and in fact, it, it really doesn't, right? Like, you, you-- in this example, I asked, "Hey, what's, what's the status of this project? Is it on track?" And as of March 16th, some model would happily pick up a, a couple of Slack messages and some slides and say, "Yep, it's green." But it's missing a lot of the context, right? It's not enough. So here's how we start building this world model. The first stage is about updating information consistently and robustly.

[7:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=449s) In this simple example, where I'm taking some slides and some decks, we are basically just looking at, you know, every month, you know, a new month comes around. We take last month's record and this month's slide, and you try to construct the sort of longitudinal record, right? And we've been doing this now for three years, right? We have a pretty large historical database of these records that now allow us to construct this robust, so-called institutional or enterprise world model. You start with the basics,

[7:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=479s) right? Like, you know, sometimes these models struggle with the most basic things, uh, but, but they're normal, right? The, the actual name of a m- of a particular project changed spontaneously, but by actually looking at that historical set of records, it can see and then infer that it actually refers to Project North Star by virtue of who was involved, by virtue of some continuity in the types of goals, those kinds of things. But more interestingly, what you want to do is you have the agentic system try to discriminate between what's being

[8:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=509s) reported and actual evidence for what it's saying. So you look at some status deck, for example, and here it's identifying that while it is green on the deck, in fact, it should be highlighted as a yellow and should be flagged as such because there's a comment someone left talking about how there's a dependency still outstanding. The main thing here is that you don't touch previous months, right? You just sort of heal in this sort of, uh, feed-forward mechanism, and what that allows you to do is to actually do some robust evals

[8:59](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=539s) down the line, right? So here's what the actual update step writes. You know, very simple. This is what it was reported. This is how it, you know, the evidence. This is where the evidence points, and this is why. And then you fold in the second component, age. The reality is every fact has a half-life. We've all seen, you know, you go back at some slides and you, you see that the go-live date for a particular project keeps slipping. It was Q2 in January, then it was

[9:29](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=569s) Q3, now it's October, right? And you actually observe this, right? By, by actually collecting all this data, you can have some pretty interesting beginnings of analytical insights. For example, in the last three years, you look at these delivery dates, and on average, half the delivery dates change within two months, and ninety percent of those change for-- into a date that is after what was originally proposed. Right? So far, so good. Pretty basic. But what you, what this forces you to do is then introduce this, this notion of, like, a, a date that, uh, is

[9:58](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=598s) associated to a particular claim so that the agent can introduce some sort of recency bias when you ask about the status of a particular project, person, or, or, or all, all of the corporate intent, let's say.

[10:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=612s) Then the observe section. The observe section here, I'm gonna pretty quickly gloss over it, but, uh, just kinda give you something very tactical. In the beginning, yeah, it was pretty seductive to just basically say, "Well, let's just YOLO this. Let's have Claude or some other model take a look at the entire corpus of things, and let's see what it does." And as it turns out, it's, it's pretty clear that a lot of, um, information is, is lost, right? It loses some attention to detail. And so what ultimately we had to

[10:42](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=642s) do was engineer a simple agentic workflow where you have a series of specialists break down the collateral of these assets, look for specific things, be reviewed by other agents, and so on. And what you see here on the, on the right-hand side is that actually the amount of insightful updates to a particular project more than doubled after doing this. So I guess the takeaway here is be intentional about how you, how you build these workflows, and don't just assume that a YOLO method is going to work out. But now

[11:12](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=672s) you get, I think, to the really interesting stuff, right? A world model should tell you what happens next, what project is about to slip or which green turns red the next month. And it turns out that you can actually do this, right? Because you can look back and establish a snapshot as of some sort of past date and es-essentially run a back test. So, for example, if we take a snapshot as of February, we can look at the future labels that came after that and see and compare what the agent said was gonna slip or flagged versus what actually

[11:42](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=702s) happened. And you can see that as the agent, um, analyzes all this information, that for statuses that were green but flagged, they were twice as likely to actually turn yellow or red. And so again, this is encouraging because we're starting to see some beginning of an ability to say, "Well, it's not predictive of exactly what's gonna go wrong, but at least it should tell you what to look at."

[12:07](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=727s) So, you know, you have 18 different cases where you have a verdict from a agent. It's flagged. But what about those where you don't? And this is where some product intuition comes in. First of all, how would you incorporate a very elegant sort of loop where you include, uh, an actual user, a stakeholder, a human? And what do you ask them, and where do you meet them, right? I, I put a non-trivial amount of work into finding out how best to actually capture this information. It's meeting the users where they are, whether it's the app they're using or

[12:37](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=757s) Slack or email, whatever. And the closer you can get to them and the simpler you can make that feedback collection mechanism, the more scalable and, and robust the process is going to be. It also matters what you ask, right? You want to understand whether, like, you're asking a fact which only fixes one output, or if you're asking about a rule. And once that an agent understands that pattern, it does a much better job of fixing every output it covers, both, uh, both now and into the future. So we have this notion

[13:06](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=786s) now of our record containing not just an element of what was inferred by the agent, but more importantly, confirmed after someone actually provided that signal. Okay, so now bottom up, right? You have meetings with, you know, really strong signal, what people say out loud. You have your Slack. You know, obviously it's constant, it's noisy, it's very close to the truth. Code changes describe what sh-shipped. And then of course, the agent sessions, which is how the sausage was made. And I think I encourage you all to

[13:36](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=816s) explore all of these disparate data sources to kind of build such a model up. With Slack, it's relatively simple, right? Like, you're just looking at every thread and trying to understand based on attachments and participants and content, the claims that you want the agent to record or update. When was this? Who was responsible, or who does this relate to? Um, what project? What r- what's, uh, what is the sort of link to the evidence and so on. With PRs, of course, it shows you what

[14:06](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=846s) changed, and it's a really nice way to see how a project is evolving. But when it comes to the actual, like, subtleties of how projects are, are, are sort of run and deployed, it's really interesting to look at the actual traces, uh, you know, for users and engineers who opt into that, right? Like, you understand how the ask was w- how it evolved over time, test failures, uh, or, you know, particular decisions from the developer, those kinds of things. Ultimately, these culminate in a pull request, but the synthesis of these

[14:36](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=876s) things, I think makes a big difference. The other gotcha, just, I just wanna share, is to just be mindful of how you think about the validity of signal coming from different so-called sensors. If you're driving down a highway, for example, and your speedometer just, you know, just goes, just disappears, it doesn't mean you're now going at zero miles an hour, right? It's just, like, a faulty signal. So, for example, here we ran into a couple of issues sometimes when the bottoms-up approach, you had some semblance of a claim saying, "Hey, this particular

[15:05](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=905s) channel, this thing is, is going quiet. What's going on?" But in reality, you know, we were treating it as a zero, when really it shouldn't have been. And so we sort of iterated over time to understand how to incorporate when we last tried to get information from that sensor, when it was last okay, and what do you do when these diverge? So now if you look at the actual record that we constructed, the agents are gonna be much more effective in understanding the true state of the world, right? It, it,

[15:35](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=935s) it, it's not green, right? It, it's, it's the go live slipped twice. The evidence says yellow. People are talking about failing tests. It's gonna be tricky. So what do we use this for, right? We use it to steer leadership conversations instead of, you know, circulating a deck to everyone and forcing everyone to just go through some slides everyone glosses over. You start with the stuff that actually, you know, matters. You start where the deck and the evidence disagree. You also produce a weekly read for managers and leaders on what stuck and what went quiet.

[16:06](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=966s) But I think also most importantly, you have the sort of beginning of a context layer that every agent can rely on. And we're using this pattern and constructing it so that it's extendable to other different use cases, right? In our client franchise, for example, we're curious about how we can build similar models to help serve our clients. We know we have some client, and we want to understand who holds what, uh, what are they interested in, how can we anticipate their needs as best as possible by folding in information from all sorts of disparate

[16:35](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=995s) internal and external sources. So a couple of things that are difficult to do, just to kind of maybe we can chat outside about your solutions, I'd be very curious to hear, are when you have a lot of, you know, for example, one client with four records and four different systems with no canonical identifier, how do you go about mapping those in a very elegant way? That's one thing that I don't think we have a clear answer to. The second thing is how can we bo- be most effective at capturing signal from stakeholders and

[17:05](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=1025s) users? Where the model is sure and the stakes are pretty low, you should just apply it automatically. When, uh, those stakes are low, but you, you know, the model is unsure, just wait for more evidence. But if the stakes are high, it really matters, like, who you ask, when you ask it, and kind of the whole mechanism there. So here's the loop again, update, age, observe, grade, and correct. And three things I want you to take home. The model that your

[17:35](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=1055s) company has, uh, should-- You should think about it as a state estimator, right? Don't assume that just, like, indexing everything and cataloging everything is enough, because by definition, these things are extremely nuanced, and this intrinsic information that lives almost between the lines is hard for even the best state-of-the-art models to capture. Grade it against what happens next, right? So be intentional and think through how you can integrate and implement evals from the get-go. And think about an

[18:05](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=1085s) elegant solution that allows your models and your, and, and your agents and your stakeholders to kind of play nice together. So your company can have a model of itself. I encourage you to start collecting now if you're not collecting already. And certainly if you have, if you're at a bigger company and like me, you had the, I mean, fortune or misfortune of dealing with thousands of decks, uh, definitely something you should start thinking about because, uh, I've definitely seen material improvements in how these agents are able to reconstruct that sort

[18:35](https://www.youtube.com/watch?v=jDwc0aXTAFg&t=1115s) of corporate ent-- uh, memory and, and apply it in all sorts of daily tasks, uh, without sort of being all confused about things. Okay. Thank you very much. I'm Joao, and I'll be outside for any questions if you want. Cheers.
