# Your Company Has No Model of Itself: Bootstrapping a World Model from Slack, Decks and Meetings

Joao Wenzel · AI Engineer NYC 2026 · October 14

[Watch recording](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23913s)

[06:38:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23913s) Okay.

[06:38:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23914s) Pleasure to be here.

[06:38:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23916s) Another round of applause for everyone at AI Engineering.

[06:38:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23919s) This has been a really, really exceptional conference so far.

[06:38:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23922s) Thank you.

[06:38:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23924s) All right.

[06:38:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23925s) So I want to begin by asking you to think about that person in your company who just

[06:38:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23929s) seems to know everything.

[06:38:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23931s) They've been through 17 reorgs.

[06:38:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23933s) They know all the gossip, all the politics.

[06:38:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23935s) and somehow they always seem to predict what's going to come out of these big strategic moves.

[06:39:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23941s) That person is a company's world model, essentially.

[06:39:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23945s) And the problem is that they don't scale and nobody really writes down what they say.

[06:39:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23951s) I'm João Wenzel with Citadel Securities

[06:39:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23954s) and I'm going to talk a little bit about how we try to approximate that with an agentic system.

[06:39:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23961s) So a little bit about Citadel Securities.

[06:39:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23962s) we build our own technology down to the silicon.

[06:39:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23966s) At this point, over two-thirds of our employees write code.

[06:39:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23970s) So naturally, we are completely AI-pilled,

[06:39:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23973s) and agents already help us with a lot of things at the organization.

[06:39:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23978s) So naturally, leadership started asking questions about strategy.

[06:39:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23982s) These harnesses are really good at producing high-quality code,

[06:39:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23986s) but can they actually help define and refine the strategy of our company?

[06:39:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23992s) So this is one of my favorite visuals I've borrowed from Tim Urban.

[06:39:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23996s) Strategy really comes down to three questions, right?

[06:39:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=23998s) Where do we stand

[06:40:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24000s) right now? What's going to happen next? And how did we get here? And to answer any of those questions, an agent has to have a really rich profile of the company that it's operating in. So this is broadly a talk about engineering an institutional memory layer. Let's call it that.

[06:40:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24021s) The agents need to have information about relationships, right?

[06:40:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24024s) Two teams that used to work together but no longer talk because there was a reorg.

[06:40:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24029s) Or a project story and why it was re-scoped several times.

[06:40:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24034s) Or the politics, right?

[06:40:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24035s) Like, who's in the room?

[06:40:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24037s) What are they saying?

[06:40:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24037s) And what are they not saying?

[06:40:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24040s) There's two ways in, right?

[06:40:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24042s) There's top-down and there's bottom-up.

[06:40:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24044s) Top down, you have the decks, you have your strategy docs,

[06:40:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24048s) your meeting and status reports,

[06:40:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24050s) and they're tidy and they're pretty clean,

[06:40:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24053s) but there's a unique human twist.

[06:40:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24055s) People generally like to infuse their thoughts and opinions

[06:40:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24058s) into these things, sometimes by design,

[06:40:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24059s) so you end up with things I call watermelon projects.

[06:41:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24062s) They're green on the outside and red on the inside.

[06:41:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24066s) Then you have, on the bottom, you have the really noisy stuff,

[06:41:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24069s) which has a surprising amount of signal,

[06:41:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24071s) but it's hard to uncover.

[06:41:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24074s) You have meeting notes, and you have slack and coat changes,

[06:41:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24078s) and even your session traces with your favorite harness.

[06:41:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24081s) And the model is where they meet.

[06:41:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24083s) The evidence is synthesized at that level.

[06:41:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24086s) Now, the evidence piece sometimes is problematic

[06:41:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24089s) because it's either partial or wrong,

[06:41:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24091s) but it turns out that in robotics,

[06:41:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24093s) you actually have the solution for this,

[06:41:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24095s) and it's called a state estimator.

[06:41:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24097s) What you do is you have some sort of belief

[06:41:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24101s) about something you care about,

[06:41:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24102s) and the longer it's been since you've seen or observed evidence

[06:41:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24106s) that supports it, the less you trust it, right?

[06:41:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24110s) The fuzzier it gets.

[06:41:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24112s) Until you have a deck or some thread or some meeting note,

[06:41:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24116s) that sort of collapses that and you now have new evidence, right?

[06:42:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24120s) And now you're more certain of it.

[06:42:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24122s) And therefore you produce the update.

[06:42:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24125s) And the things that I care about when producing an update

[06:42:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24128s) or that the agent should care about, there's three things.

[06:42:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24131s) There's how long has it been since some signal has been received.

[06:42:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24135s) There's where did that signal come from, and is it still trustworthy?

[06:42:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24140s) And then does it broadly agree with other sensors?

[06:42:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24143s) There's a lot of parallels here with self-driving, for example, that I'll come back to.

[06:42:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24148s) So let's start at the top.

[06:42:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24149s) A very simple example is something like a slide that is produced already by someone in BizOps,

[06:42:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24156s) and they produce a monthly status deck.

[06:42:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24158s) and the question is, how do you get data out of this slide?

[06:42:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24162s) Many of you, I'm sure, have tried that.

[06:42:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24164s) It's deceptive, actually, because let's say you try to parse the file.

[06:42:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24169s) You unpack the PowerPoint file and you get an XML

[06:42:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24172s) and all of a sudden you get into this really interesting conceptual question.

[06:42:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24175s) What is green? What is the hex code for green?

[06:43:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24180s) And things break down when it comes to this type of script.

[06:43:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24183s) Then the solution is to say, well, I'm just going to take a screenshot of it

[06:43:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24186s) and throw it at a vision model.

[06:43:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24188s) But at that point, even if it's pretty high res, you're dealing with someone who copy-pasted

[06:43:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24192s) a screenshot back into a presentation, and essentially you have Claude trying to squint

[06:43:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24197s) and trying to figure out if it's a 31 or an 81.

[06:43:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24200s) And so the theme of the talk really is about elegance comes from just synthesizing things

[06:43:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24206s) and adding more information.

[06:43:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24208s) And in this example, it's very basic, but you get the true picture by taking both the

[06:43:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24213s) XML and the output of a vision model.

[06:43:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24216s) Now, that's pretty basic, right?

[06:43:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24218s) But the issue is that you want to establish some continuity over time, right?

[06:43:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24223s) Like this collateral that your company accrues is very much sort of always building up.

[06:43:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24228s) And so in this particular example, you have a project that just spontaneously changed names back in March.

[06:43:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24235s) And so, you know, you have to establish that sort of longitudinal traceable audit layer

[06:44:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24240s) as you create this shared context layer about the sort of institutional memory of your company.

[06:44:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24247s) And the irony, there's two things.

[06:44:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24249s) The irony here is that even if you do this really well,

[06:44:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24252s) you're still missing half the picture, right?

[06:44:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24254s) You're looking at a deck and your Slack says otherwise

[06:44:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24257s) because that was five days ago.

[06:44:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24259s) And now you have someone on Slack saying,

[06:44:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24260s) hey, by the way, I think we're on track for this.

[06:44:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24263s) But then someone replies with a picture of a dashboard

[06:44:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24265s) and it's no longer the case.

[06:44:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24266s) So things just start breaking down.

[06:44:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24268s) And so we need to do a better job

[06:44:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24271s) of integrating all these things, right?

[06:44:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24274s) Your company has no model of itself.

[06:44:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24277s) So I'm proposing just a brief plan here.

[06:44:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24280s) The idea is you take the three things that I described earlier

[06:44:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24283s) about the state estimator.

[06:44:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24284s) You have the update, age, and observe stages,

[06:44:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24288s) and then you have grade.

[06:44:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24290s) The grade stage is basically comparing what you observed

[06:44:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24293s) versus what you expected.

[06:44:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24295s) And then the correct stage is basically a way for people

[06:44:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24299s) to allow the system to get new feedback

[06:45:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24301s) and steer the behavior of the system.

[06:45:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24305s) At Citadel Securities, the system that we've built has agents at every part of this particular

[06:45:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24311s) loop.

[06:45:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24312s) But the key thing is that the model is proposing these changes, and then you have code and

[06:45:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24316s) people deciding how the records are updated.

[06:45:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24319s) And of course you're probably asking yourself, why not just RAG?

[06:45:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24323s) Probably RAG would do the trick.

[06:45:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24324s) And in fact, it really doesn't.

[06:45:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24327s) In this example, I asked, hey, what's the status of this project?

[06:45:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24330s) Is it on track?

[06:45:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24331s) And as of March 16th, some model would happily pick up a couple of Slack messages and some

[06:45:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24337s) slides and say, yep, it's green.

[06:45:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24339s) But it's missing a lot of the context, right?

[06:45:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24341s) It's not enough.

[06:45:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24343s) So here's how we start building this world model.

[06:45:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24346s) The first stage is about updating information consistently and robustly.

[06:45:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24351s) In this simple example, where I'm taking some slides and some decks, we are basically just

[06:45:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24356s) looking at every month.

[06:45:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24358s) A new month comes around, we take last month's record and this month's slide,

[06:46:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24363s) and you try to construct a sort of longitudinal record.

[06:46:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24367s) And we've been doing this now for three years.

[06:46:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24369s) We have a pretty large historical database of these records

[06:46:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24373s) that now allow us to construct this robust so-called institutional or enterprise world model.

[06:46:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24379s) You start with the basics.

[06:46:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24382s) Sometimes these models struggle with the most basic things, but they're normal.

[06:46:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24386s) The actual name of a particular project changed spontaneously,

[06:46:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24391s) but by actually looking at that historical set of records,

[06:46:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24394s) it can see and infer that it actually refers to Project North Star

[06:46:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24398s) by virtue of who was involved, by virtue of some continuity in the types of goals,

[06:46:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24403s) those kinds of things.

[06:46:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24405s) But more interestingly, what you want to do is you have the agentic system

[06:46:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24409s) try to discriminate between what's being reported and actual evidence for what it's saying.

[06:46:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24414s) So you look at some status deck, for example, and here it's identifying that while it is green on the deck, in fact it should be highlighted as a yellow and should be flagged as such because there's a comment someone left talking about how there's a dependency still outstanding.

[06:47:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24429s) The main thing here is that you don't touch previous months, right?

[06:47:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24433s) You just sort of heal in the sort of feed-forward mechanism.

[06:47:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24437s) And what that allows you to do is to actually do some robust evals down the line, right?

[06:47:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24443s) So here's what the actual update step writes.

[06:47:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24446s) Very simple.

[06:47:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24447s) This is what it was reported.

[06:47:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24449s) This is where the evidence points, and this is why.

[06:47:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24454s) And then you fold in the second component, age.

[06:47:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24458s) The reality is every fact has a half-life.

[06:47:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24460s) We've all seen, you know, you go back at some slides,

[06:47:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24463s) and you see that the go-live date for a particular project keeps slipping.

[06:47:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24468s) It was Q2 in January, then it was Q3, now it's October.

[06:47:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24472s) And you actually observe this.

[06:47:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24475s) By actually collecting all this data,

[06:47:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24476s) you can have some pretty interesting beginnings of analytical insights.

[06:48:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24480s) For example, in the last three years, you look at these delivery dates,

[06:48:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24484s) and on average, half the delivery dates change within two months,

[06:48:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24487s) and 90% of those change into a date that is after what was originally proposed.

[06:48:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24492s) So far, so good. Pretty basic.

[06:48:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24494s) But what this forces you to do is then introduce this notion of a date that is associated to a particular claim

[06:48:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24502s) so that the agent can introduce some sort of recency bias when you ask about the status of a particular project, person, or the corporate intent, let's say.

[06:48:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24514s) Then the observe section.

[06:48:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24516s) The observe section here, I'm going to pretty quickly gloss over it, but just kind of give you something very tactical.

[06:48:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24522s) In the beginning, I was pretty seductive to just basically say, well, let's just YOLO this.

[06:48:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24526s) Let's have Claude or some other model take a look at the entire corpus of things, and let's see what it does.

[06:48:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24533s) And as it turns out, it's pretty clear that a lot of information is lost.

[06:48:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24539s) It loses some attention to detail.

[06:49:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24542s) And so what ultimately we had to do was engineer a simple agentic workflow

[06:49:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24547s) where you have a series of specialists break down the collateral of these assets

[06:49:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24551s) look for specific things, be reviewed by other agents, and so on.

[06:49:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24556s) And what you see here on the right-hand side

[06:49:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24559s) is that actually the amount of insightful updates

[06:49:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24562s) to a particular project more than doubled after doing this.

[06:49:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24566s) So I guess the takeaway here is be intentional

[06:49:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24568s) about how you build these workflows

[06:49:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24570s) and don't just assume that a YOLO method is going to work out.

[06:49:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24574s) But now you get, I think, to the really interesting stuff.

[06:49:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24576s) A world model should tell you what happens next.

[06:49:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24579s) What project is about to slip,

[06:49:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24581s) or which green turns red the next month.

[06:49:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24583s) And it turns out that you can actually do this, right?

[06:49:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24586s) Because you can look back and establish a snapshot

[06:49:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24589s) as of some sort of past date

[06:49:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24591s) and essentially run a backtest.

[06:49:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24593s) So for example, if we take a snapshot as of February,

[06:49:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24595s) we can look at the future labels that came after that

[06:49:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24599s) and see and

[06:50:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24600s) compare what the agent said was going to slip or flagged versus what actually happened.

[06:50:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24605s) And you can see that as the agent analyzes all this information, that for statuses that were green but flagged, they were twice as likely to actually turn yellow or red.

[06:50:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24616s) And so, again, this is encouraging

[06:50:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24618s) because we're starting to see some beginning

[06:50:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24620s) of the inability to say,

[06:50:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24622s) well, it's not predictive of exactly what's going to go wrong,

[06:50:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24625s) but at least it should tell you what to look at.

[06:50:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24628s) So, you know, you have 18 different cases

[06:50:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24631s) where you have a verdict from an agent that's flagged,

[06:50:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24634s) but what about those where you don't?

[06:50:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24636s) And this is where some product intuition comes in.

[06:50:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24638s) First of all, how would you incorporate

[06:50:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24639s) a very elegant sort of loop

[06:50:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24641s) where you include an actual user, a stakeholder, a human?

[06:50:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24646s) What do you ask them and where do you meet them?

[06:50:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24649s) I put a nontrivial amount of work into finding out how best to actually capture this information.

[06:50:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24655s) It's meeting the users where they are, whether it's the app they're using, Slack, email,

[06:51:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24660s) whatever.

[06:51:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24661s) The closer you can get to them and the simpler you can make that feedback collection mechanism,

[06:51:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24665s) the more scalable and robust the process is going to be.

[06:51:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24669s) It also matters what you ask.

[06:51:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24670s) You want to understand whether you're asking a fact which only fixes one output,

[06:51:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24676s) or if you're asking about a rule.

[06:51:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24678s) And once an agent understands that pattern,

[06:51:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24680s) it does a much better job of fixing every output it covers,

[06:51:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24683s) both now and into the future.

[06:51:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24686s) So we have this notion now of our record containing not just an element

[06:51:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24692s) of what was inferred by the agent,

[06:51:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24694s) but more importantly confirmed after someone actually provided that signal.

[06:51:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24699s) Okay, so now bottom up, right?

[06:51:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24702s) You have meetings with really strong signal, what people say out loud.

[06:51:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24705s) You have your Slack.

[06:51:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24707s) Obviously, it's constant.

[06:51:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24708s) It's noisy.

[06:51:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24709s) It's very close to the truth.

[06:51:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24711s) Code changes describe what shipped.

[06:51:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24713s) And then, of course, the Asian sessions, which is how the sausage was made.

[06:51:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24716s) And I think I encourage you all to explore all of these disparate data sources to kind

[06:52:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24720s) of build such a model up.

[06:52:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24722s) With Slack, it's relatively simple, right?

[06:52:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24725s) like you're just looking at every thread and trying to understand,

[06:52:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24727s) based on attachments and participants and content,

[06:52:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24731s) the claims that you want the agent to record or update.

[06:52:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24735s) When was this? Who was responsible?

[06:52:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24738s) Or who does this relate to? What project?

[06:52:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24741s) What is the sort of link to the evidence? And so on.

[06:52:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24746s) With PRs, of course, it shows you what changed.

[06:52:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24748s) And it's a really nice way to see how a project is evolving.

[06:52:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24752s) But when it comes to the actual subtleties of how projects are run and deployed, it's

[06:52:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24759s) really interesting to look at the actual traces for users and engineers who opt into that.

[06:52:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24764s) You understand how the ask evolved over time, test failures, or particular decisions from

[06:52:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24772s) the developer, those kinds of things.

[06:52:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24775s) Ultimately, these culminate in a pull request, but the synthesis of these things, I think,

[06:52:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24778s) makes a big difference.

[06:53:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24780s) The other gotcha I just want to share is to just be mindful of how you think about the

[06:53:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24786s) validity of signal coming from different so-called sensors.

[06:53:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24789s) If you're driving down a highway, for example, and your speedometer just disappears, it doesn't

[06:53:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24796s) mean you're now going at zero miles an hour.

[06:53:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24798s) It's just like a faulty signal.

[06:53:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24800s) So for example, here we ran into a couple of issues sometimes when the bottoms-up approach,

[06:53:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24804s) you had some semblance of a claim saying,

[06:53:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24807s) hey, this particular channel, this thing is going quiet, what's going on?

[06:53:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24811s) But in reality, we were treating it as a zero,

[06:53:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24813s) and really it shouldn't have been.

[06:53:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24814s) And so we sort of iterated over time to understand

[06:53:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24818s) how to incorporate when we last tried to get information from that sensor,

[06:53:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24823s) when it was last okay, and what do you do when these diverge?

[06:53:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24828s) So now, if you look at the actual record that we constructed,

[06:53:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24832s) the agents are going to be much more effective

[06:53:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24834s) in understanding the true state of the world.

[06:53:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24837s) It's not green.

[06:53:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24839s) It's the go-live, slip-twice, the evidence says yellow,

[06:54:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24842s) people are talking about failing tests.

[06:54:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24844s) It's going to be tricky.

[06:54:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24846s) So what do we use this for?

[06:54:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24848s) We use it to steer leadership conversations.

[06:54:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24850s) Instead of circulating a deck to everyone

[06:54:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24853s) and forcing everyone to just go through some slides

[06:54:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24855s) everyone glosses over,

[06:54:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24856s) you start with the stuff that actually matters.

[06:54:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24859s) You start where the deck and the evidence disagree.

[06:54:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24862s) You also produce a weekly read for managers and leaders

[06:54:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24865s) on what stuck and what went quiet.

[06:54:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24867s) But I think also, most importantly,

[06:54:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24869s) you have the sort of beginning of a context layer

[06:54:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24871s) that every agent can rely on.

[06:54:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24874s) And we're using this pattern and constructing it

[06:54:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24876s) so that it's extendable to other different use cases.

[06:54:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24879s) Right in our client franchise, for example,

[06:54:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24881s) we're curious about how we can build similar models

[06:54:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24883s) to help serve our clients.

[06:54:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24884s) We know we have some client,

[06:54:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24886s) and we want to understand who holds what,

[06:54:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24889s) what are they interested in,

[06:54:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24891s) how can we anticipate their needs as best as possible

[06:54:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24894s) by folding in information

[06:54:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24895s) from all sorts of disparate internal and external sources.

[06:55:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24900s) So a couple of things that are difficult to do,

[06:55:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24902s) just to kind of, maybe we can chat outside about your solutions,

[06:55:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24905s) I'd be very curious to hear,

[06:55:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24907s) are when you have a lot of, you know, for example,

[06:55:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24910s) one client with four records and four different systems

[06:55:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24912s) with no canonical identifier,

[06:55:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24913s) how do you go about mapping those in a very elegant way?

[06:55:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24918s) That's one thing that I don't think we have a clear answer to.

[06:55:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24921s) The second thing is, how can we be most effective at capturing signal from stakeholders and users?

[06:55:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24928s) Where the model is sure, and the stakes are pretty low, you should just apply it automatically.

[06:55:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24933s) When those stakes are low, but the model isn't sure, just wait for more evidence.

[06:55:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24938s) But if the stakes are high, it really matters who you ask, when you ask it,

[06:55:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24943s) and kind of the whole mechanism there.

[06:55:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24946s) So here's the loop again.

[06:55:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24949s) Update, age, observe, grade, and correct.

[06:55:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24953s) And three things I want you to take home.

[06:55:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24955s) The model that your company has,

[06:55:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24959s) you should think about it as a state estimator, right?

[06:56:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24962s) Don't assume that just like indexing everything

[06:56:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24965s) and cataloging everything is enough

[06:56:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24967s) because by definition, these things are extremely nuanced

[06:56:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24971s) and this intrinsic information that lives almost between the lines

[06:56:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24974s) is hard for even the best state-of-the-art models to capture.

[06:56:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24978s) Grade it against what happens next, right?

[06:56:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24980s) So be intentional and think through how you can integrate

[06:56:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24983s) and implement evals from the get-go.

[06:56:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24985s) And think about an elegant solution that allows your models

[06:56:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24989s) and your agents and your stakeholders to kind of play nice together.

[06:56:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24996s) Your company can have a model of itself.

[06:56:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=24999s) I encourage you to start collecting now

[06:56:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25000s) if you're not collecting already.

[06:56:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25002s) And certainly if you have, if you're at a bigger company

[06:56:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25004s) and like me you had the fortune or misfortune

[06:56:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25007s) of dealing with thousands of decks,

[06:56:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25009s) definitely something you should start thinking about

[06:56:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25011s) because I've definitely seen material improvements

[06:56:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25014s) in how these agents are able to reconstruct

[06:56:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25016s) that sort of corporate memory

[06:56:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25019s) and apply it in all sorts of daily tasks

[06:57:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25022s) without sort of being all confused about things.

[06:57:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25026s) Okay, thank you very much.

[06:57:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25027s) I'm João and I'll be outside for any questions if you want.

[06:57:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25030s) Cheers.

[06:57:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=25030s) Thank you.