AI Engineer World's Fair 2026
Notion's Token Town
Read the talk
Notion’s Token Town: Building AI Systems You Can Afford to Run
Sustainable AI products require more than cheaper tokens: they need task-specific model choices, credible alternatives, deterministic execution and a durable place for people and agents to collaborate.
From a talk by Sarah Sachs
How do you build an AI company without becoming AI poor?
How do you turn enthusiasm for AI into a product—and an operating company—that can afford its own usage? That is the opening problem of Token Town. Sarah Sachs leads Notion’s AI engineering teams and negotiates its AI contracts. Her team’s Anna Wintour comparison is partly a joke about taste, but it also captures a practical responsibility: choose among suppliers without letting their offerings define the company’s product.
The work extends across the company, not just the people pictured at a launch. Sachs shows rising AI usage during 2026 as the backdrop to a broader change in Notion’s role. A durable system of record once primarily connected people with other people. Now it must also support people working with agents, and agents working with one another. Growth makes that collaboration valuable; it also makes the economics of every repeated operation consequential.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From drafting an email to running a system
The first step is familiar: ask ChatGPT to draft an email telling a landlord why you should not pay for repainting, then copy the answer into your email client. The model helps you think and write, but you still carry the result into the world. As AI gains responsibility, the unit of work changes:
| Stage | What AI does | What remains outside it |
|---|---|---|
| Thought partner | Drafts the landlord email | Moving and sending the result |
| Assistant | Executes an individual requested task | Deciding which task comes next |
| Teammate | Performs a repetitive process | Connecting separate processes |
| System | Connects processes into operating workflows | Governing the whole system |
Early Notion AI fit the assistant stage: it saved employee time, but its capabilities were bounded by individual human requests. The teammate stage extends that into repeatable processes. The system stage requires those processes to interface with one another.
Sachs says 88% of people cannot get past AI as an assistant, without identifying the study or population behind that figure. Notion’s explanation is that data remains siloed and collaboration lacks a durable record. If one process cannot reliably find the decisions, context and results of another, adding more agents does not automatically produce a functioning system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An unchanged token price can hide a larger bill
Trying to buy your way from isolated tasks to an operating system can become extravagantly expensive. Sachs illustrates the mismatch with a blowtorch lighting a large cigar: the machinery works, but it is disproportionate to the job. Cost is a structural barrier to scale, not merely a procurement detail to revisit after the product succeeds.
Two upgrade scenarios make the problem concrete:
| Upgrade | What changes | Economic consequence |
|---|---|---|
| Reasoning-model upgrade | Sachs reports unchanged per-token pricing but three times as many output tokens. | The output-token portion of the bill triples for that usage. |
| Successor-model upgrade | Sachs reports a 40% price increase and predecessor deprecation within four months. | Keeping the old price is no longer a durable option. |
These are unnamed scenarios Sachs says Notion has faced; the workloads and billing details are not specified. The first exposes the difference between a unit price and the quantity consumed. The second combines a higher price with a deadline that weakens the buyer’s ability to stay put. Automatically upgrading without corresponding revenue growth passes the resulting loss to customers or investors.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The supplier may also be your competitor
Large enterprises can hire consulting teams and develop internal expertise to navigate these changes. Smaller companies—the Fortune five million in Sachs’s framing—usually cannot bring equivalent negotiating power. Her aim is to translate lessons learned with Notion’s traffic volume into practices that remain useful without that volume.
The awkward starting point is that a model supplier may also sell a competing first-party product. That supplier produces the tokens; the application developer buys them at a markup and must then charge enough to support another business on top. Sachs argues that this resale structure is not, by itself, defensible value. Dependence on one provider makes the problem worse because it removes the option to leave.
A subscription-versus-API comparison posted by Dylan at SemiAnalysis illustrates her concern about the gap between first-party access and the economics offered to developers. The talk does not supply the numerical comparison. The product implication is nevertheless clear: a business built principally around reselling access is exposed to a supplier that controls both its input price and a competing customer offer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match capability to the customer’s task
The defensible work sits above token resale: understand customers, build data flywheels, and use that knowledge to decide when capability, price or latency matters most. Compelling interfaces and orchestration can make purchased inference worth paying for. The most capable model is often also slow, and many customer tasks do not require its full capability. For an applied AI company, the goal is usually the best product using multiple models, rather than training the best model. Sachs’s formulation is: “Bet on the frontier, not on the lab.”
She connects this to an economic split between frontier and everyday usage. The Citadel Securities analysis Tokenomics closely matches the memo she describes: simpler models may provide a more cost-effective path to productivity. But the boundary is product-specific. A capability can be scarce for one workflow and already more than sufficient for another.
Notion’s examples put that distinction into operational terms:
- Large-scale data analysis: Sachs says Notion recommends Opus when the work warrants frontier capability.
- Inbox triage: Paying for that same capability can waste both the customer’s money and the application’s margin.
The routing decision begins with the task and its traffic pattern, not the newest model name.
Frontier tasks still matter. Sachs characterizes the leading providers as an oligopoly racing toward difficult capabilities, then explains why their pricing may not reflect the incremental value of those capabilities to your product. A leader can charge a premium for work that previously could not be done. A follower may only need to undercut the leader slightly—her illustrative example is a dollar per million tokens—to attract the rest of the market. Like adjacent gas stations, competitors can cluster their offers rather than produce a smooth price curve for every level of customer need. Your product team must define task complexity; the price list cannot do it for you.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the ability to switch real
The provider best equipped for a difficult task changes as new models arrive. Marketing a product around one lab, or accepting a discount that effectively binds the product to that lab, can leave customers behind when another provider takes the lead. Interoperability has an engineering cost, but Sachs treats the ability to walk away as a source of negotiating power that can outweigh the discount.
Notion keeps multiple models available so that a pricing change does not become an existential product problem. Sachs reports that Notion’s Auto model handles about 75% of its traffic; she does not specify the measurement window or traffic denominator. State-of-the-art models remain available, and customers can also choose among them. This is the practical meaning of Notion’s AI Switzerland approach: model choice exists inside the product rather than being fixed by one supplier relationship.
The operating playbook requires more than adding another model to a menu:
- Build interoperability into the harness. Switching models mid-transcript can invalidate caches. Supporting multiple models does not require switching within every thread, but the architecture must make a change possible.
- Evaluate complete trajectories. Compare capability, cost and elapsed time across the full task. Sachs uses Parallel’s web search as an example: an individual call may not look cheapest or fastest, while an entire search trajectory exposes a different set of trade-offs.
- Keep switching practical. Product-specific evaluations provide the evidence needed to change models quickly and repeatedly.
- Bring suppliers useful evidence. Use-case evaluations and early-access feedback have value to frontier labs. Sachs describes exchanging that expertise instead of relying on extraordinarily large purchasing commitments.
The evaluation unit matters: a cheap call that leads to more work can be a poor choice for a finished task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Open weights create a credible alternative
Sachs’s preference is explicit: a discount is not worth losing optionality. Open-weight models expand that optionality for moderate tasks, and reinforcement learning can extend the range of work they handle. Their importance is both technical and commercial. A credible alternative can reduce customer costs while placing pressure on prices that would otherwise be set among a few frontier providers.
Sachs reports that Kimi K2.6 outperformed GPT-5.2, but does not give the task or evaluation conditions for that comparison. She also points to GLM-5.2 as another sign of progress. The useful engineering question is broader than those model names: can an open-weight model already do your work without additional reinforcement learning? Do not assume its only role is supervised fine-tuning for small tasks.
Answer that question with your own system’s tool errors and latency requirements, rather than external benchmark rankings alone. Sachs deliberately shows an older internal benchmark to emphasize the evaluation method rather than a current leaderboard. A conceptual slide credited to Philip at Baseten then makes the longer-term point: open models do not have to lead the frontier to cross a capability threshold your product needs. Its labeled thresholds progress from tab completion to code generation to a reasoning agent.
Sachs predicts that tasks adequately served today will probably be covered by open weights within six months. That is a planning forecast: build the ability to adopt an alternative before the alternative becomes attractive enough to force an urgent migration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Execute deterministic work as code
Some jobs do not need a different model; they need no inference at execution time. Sachs introduces Notion’s recently launched Workers under the principle of CPUs over GPUs. Converting a CSV into a PDF is a discrete program. Invoking Notion operations through a CLI does not inherently require an LLM to mediate each tool call. A deterministic SQL query should run in the database.
For example, counting open tasks by team is ordinary SQL. With a tasks table containing team_id and status, the operation can be expressed directly:
sql
SELECT
team_id,
COUNT(*) AS open_task_count
FROM tasks
WHERE status = 'open'
GROUP BY team_id
ORDER BY team_id;
An agent may help decide that this is the needed query, but repeatedly performing the aggregation does not require asking a model to reason through the rows. The distinction is between deciding what work to do and executing work whose rules are already known.
Governance adds another kind of value beyond inference: visibility into who uses data, maintainability and control. Model optionality also expands the choices a product can offer its customers. Sachs points to Notion’s governance interface without detailing its controls, then closes the economics portion around architecture, open weights and value that transcends tokens.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Autonomy makes security and persistence central
Looking beyond Token Town, Sachs expects security rather than raw capability to define the next six months of product work. Simon Willison’s lethal trifecta describes the dangerous combination:
- Private-data access: the agent can read information that should remain protected.
- Untrusted content: ingestion, MCP or email can introduce material an attacker controls.
- External communication: the agent can send information outside the protected environment, including through payloads in web searches.
Together, these capabilities expose a route from malicious instructions in incoming content to private information leaving the system. More autonomy leaves more of that exposure unsupervised. Managing it is part of the product’s value, not an optional layer around model capability.
Sandboxes and computer environments connect the security discussion back to determinism and token economics. Multi-agent orchestration adds another set of questions: what can each agent see, what can it do, and what survives after it finishes? Sachs singles out the persistence of enterprise knowledge as an under-discussed requirement.
Without reliable coordination, a software factory can turn into an engineering team babysitting the factory. Sachs says Notion’s own efforts began that way. Orchestration is difficult because the system must preserve useful context and advance work across boundaries, not merely start several agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shared task document coordinates people and agents
A promotional slide featuring T-Pain introduces an openly acknowledged product pitch, followed by the concrete workflow. Sachs starts with a task in a Notion document and a Manage Agent capability she describes as launched that day. Claude Agent can inspect the task and scope the required work.
The important step is what happens when the task is not ready to implement. The agent begins populating a specification but asks a question about missing requirements. Instead of treating the response as the end of a chat, Sachs brings a teammate into the same document, tagging MJ, the product manager. The document is an active collaboration surface where a question can move from an agent to a person without losing the task around it.
MJ does not know the answer in this example either. That opens another handoff: Claude Code may be useful for implementation, while Decagon agents are better suited to collecting customer-voice evidence. Sachs asks those agents for the relevant data, then advances through part of the flow. A team lead replies in the shared document, supplying the human input needed to proceed. Claude can then create the pull request.
Claude reports that it has put up a PR. The selected frame shows the customer research and a human instruction to build above the Claude Agent reply, which includes a pull-request link and a Code explainer link. Implementation is now a concrete artifact in the ongoing discussion, but Sachs does not treat that as the end: she asks Codex to review it.
Sachs reports that Codex finds two issues in the demonstrated pull request. The workflow therefore crosses both human and provider boundaries: an agent scopes, people clarify, a specialist supplies evidence, a coding agent implements, and another agent reviews. The shared task record coordinates those contributions without requiring the whole factory to commit to one lab.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a demonstration to everyday operations
Sachs follows the demonstration with an internal example shared with Rajiv’s permission. She says almost all Notion polish work and larger feedback of the illustrated kind is coordinated through its software factories: work is routed to the right teams, and coding agents take the first implementation attempt. She also cites Vercel using the workflow from staging through shipping and closing.
Sachs reports over three minutes saved on a given customer task, without specifying the task, baseline or measurement conditions. The larger operational point is that repeated savings depend on knowing which models each task requires and preserving the option to change them. Factory economics emerge from the whole workflow, including coordination and review, rather than from a model’s token price alone.
That leaves product builders with an ongoing responsibility in a young, opaque and fast-moving market. Sachs credits communities such as AI Engineer with making these trade-offs discussable in public. Customers depend on application teams to scrutinize model purchasing and product decisions on their behalf—even as the available capabilities, prices and suppliers keep changing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Frank Flight examines inference costs, physical constraints and the economic split between frontier and everyday AI workloads.
Simon Willison explains how private-data access, untrusted content and external communication combine to create prompt-injection risks.
Official model weights, deployment examples, licensing terms and task-specific evaluation results for Moonshot AI's Kimi K2.6.
Parallel's web-search product for AI applications, with access to documentation and a search playground.
Further reading
Notion introduces Workers, its CLI and external-agent building blocks, with links to developer setup resources.
Read the complete timestamped transcript
- 0:00
[upbeat music] Okay.
- 0:22
Hello. Okay, before I get started, you guys, this is a huge keynote room. Can everyone, like, come forward? 'Cause I'm talking to, like, four empty rows and dispersed people.
- 0:31
Do me a favor. I'm spending thirty minutes telling you all of our secrets. I can see you still. Thank you, thank you, thank you, thank you. We're just gonna chat.
- 0:39
It's a giant room, and there's five hundred of us. This room is way larger than that. Thank you. Honestly, I knew you guys had it in you. It's really not so hard.
- 0:49
Thank you. I also sit in the back. I also work during talks. I get it.
- 0:55
I totally get it. I did all day, but not for me. Okay. I'm gonna start, but I'm gonna still point at you if you're in the back, like you. [clicks tongue]
- 1:02
Okay. I'm Sarah. I, um, lead our engineering teams for AI at Notion. Um, welcome to my talk. Um, it's about Token Town. How do you go from-- not go from AI pilled to AI poor, okay?
- 1:19
Um, I know that today's all about software factories. We're gonna talk about that, but we're gonna talk about how to do it sustainably. This is me. This is on my first day at Notion in a very sweaty subway.
- 1:30
Um, like I said, I lead our AI teams at Notion. Um, and I negotiate AI contracts for a living. My team jokes I act like Anna Wintour, so this is a nice, a nice image of me with AI Anna Wintour hair after a press article referred to me that externally.
- 1:46
Um, and that's kind of the idea, right? Uh, how do you think about negotiating between different vendors, um, making sure that you maintain taste for your company? I don't do it alone.
- 1:58
Um, this is launch day at one of our recent launches. This is just a subset. Any good engineering manager points out that we have a whole company of people building this.
- 2:05
I'm just the one that gets to come talk to you about it.
- 2:08
So we've been building a lot. Um, this is, um, an example of our AI usage, um, just in twenty twenty-six. Um, and we've been really proud of how we've been able to grow that usage, and I'm gonna talk to you about how you can build an AI-native product and an AI-native company.
- 2:25
Um, but this is just to give me some credit that, that we're doing it kind of well. Okay. So for those of you that don't know, Notion's always been that durable system of record.
- 2:34
It's always been the place where you can collaborate with your peers. Um, but today, that point of collaboration is a little bit different. It's not just humans. Notion's always been the place for collaboration, and today that collaboration happens between humans and agents, humans and humans, agents and agents.
- 2:52
And we like to think about AI transformations going through this journey, and I'm sure some of you are looking at this slide and wondering where you are. AI as a thought partner is when we all started tinkering.
- 3:03
We all started just going to the very first version of ChatGPT on Thanksgiving when it came out three years ago? Four years ago? And we started saying, like, "How can I send this email to my landlord to say that I shouldn't pay for repainting?"
- 3:17
Right? [chuckles] And then we'd copy-paste it, enter it into our email. Eventually, we started getting to a place where we could use AI like an assistant. AI was able to maybe execute individual tasks.
- 3:29
That's how Notion AI really took off in the beginning. Um, and it was able to save employee time, but it functionally was limited in its capabilities based on what humans asked it to do.
- 3:38
AI as teammates is what we were really excited to launch almost a year ago now. Um, but this is true in many products where you can re-r- do repetitive work and think about a process and have AI do that process.
- 3:51
What I think is really interesting is when AI actually becomes that critical workflow where processes are interfacing with each other and you have entire systems running. How many of you guys feel like you have AI as a system down?
- 4:03
Aren't you sad you came up now? I'm kidding. Great. None of you. Exactly. We have found that no one has figured out how to do this well. Eighty-eight percent of people can't even get past AI as an assistant.
- 4:16
And why is that? We have a thesis at Notion it's because there's too much siloed data and not a durable system of record for that point of collaboration, and we believe that for your software factory to work, for your company to work, and for your systems to work, you need that durable system of record, and that is
- 4:33
Notion's mission. So doing that is expensive. Um, you see a lot of companies that try and commit themselves to this vision, and these are just a series of headlines all within a week of how that's painful.
- 4:49
So you can put all of your money into a process to try and make a system, and you end up feeling like this,
- 4:56
right? You end up using a blowtorch to light what is actually a large cigar, but you kinda get the idea.
- 5:04
Cost is a structural barrier to entry. It makes it hard for you to serve products. It makes it hard for you to build factories. And it is ultimately, I would posit, one of the largest reasons why things do not happen at scale successfully today.
- 5:18
And I would argue for anyone working at an applied AI company, it's something for them to be really familiar with to understand the trade-offs that they're making to build durable and exciting and enlightening product for their customers.
- 5:32
But that's not really how the market is today, right? I'm not gonna name names here, but you guys have search engines. You can figure it out. Exhibit A, a reasoning model gets upgraded.
- 5:43
Amazing. The per-token pricing is the same. What's not to love? You try it out. It uses three times as many output tokens, right? Exhibit B, a model gets upgraded, but it has an entire new digit, right?
- 5:59
Whatever marcation system that model family likes, it's brand new. It's forty percent more than its predecessor, which is being deprecated in the next four months.
- 6:07
These are real scenarios that we face at Notion. All of you are nodding because these are common, pretty much monthly now. But here's the problem. Are you growing forty percent in that time period?
- 6:20
Are you making thirty-three x more revenue? No. So how do you navigate the system? If you just auto-upgrade your model and everything that you're doing, you're, you're giving someone a bad deal, either your customers or your investors, depending on how you charge and where you get your money.
- 6:36
Neither are good. Fortune five million companies have the capability to navigate this. They can hire large consulting teams, have durable teams on their own, and build expertise on how to navigate these trade-offs.
- 6:51
Um, most people don't. Everyone else has no ability to negotiate with leverage, and they're stuck in these scenarios, right?
- 6:59
Part of my job as that Anna Winter joke is to think about advocating for the Fortune five million, the non-Fortune five hundred companies that don't have the mass to have leverage and negotiate, but need to think about how, and I'm gonna share some of the lessons that I've learned when I have kind of large amounts of traffic
- 7:17
behind me that I think scale to those who don't.
- 7:22
Um, this is probably less of a secret now than it was when I started giving talks like this, uh, maybe four months ago. Um, your supplier is your competitor.
- 7:33
Uh, I know very few people who have convinced me that that's not true.
- 7:38
Um, you will always be getting a bad deal on tokens with someone who builds them natively, right? Sometimes the cost of goods served is extremely different. You're basically-- They're serving a first-party product, and then you're buying those tokens at a huge surcharge, and then selling them again at another surcharge.
- 7:54
Um, that's not really value you can defend. You're getting a really bad deal. And if you tie yourself to one provider, you have no exit. If you build an AI product that you're selling with this structure, you are crossing your fingers and hoping that you are a viable business.
- 8:11
I do not encourage that. This is really interesting. Dylan in Semia analysis posted this. I think it's-- it says eight hours ago. It wasn't at this point. It was probably a month ago.
- 8:21
Um, they purchased a subscription plan, and they just highlighted, right, how different what frontier labs charge customers for first-party products are versus what they sell. It's a bad deal.
- 8:34
Don't play this game. Or try and let me know how you win. I don't recommend.
- 8:41
Think about everyone else. Think about what that structure means and where you have expertise.
- 8:48
I don't think that that's winning on the token economics. I think it's about product. It's about building data flywheels and understanding your customers better than anyone else, understanding when you need capability, when you need low price, when you need latency improvements.
- 9:06
I promise you, you don't always need what is usually the slowest but the most capable model out there. And then build compelling UI and orchestration, and I'll show you some examples of that, to justify the cost on the bad deal tokens that you do resell.
- 9:24
The job is not to train-- I mean, some of you might be training the best model, and I'd love to serve it and come talk to me afterwards, but most of you are not doing that.
- 9:31
Stop trying to win that game and think about the best product that uses many models. Help your customers. Help your team. Bet on the frontier, not on the lab, and we'll talk about what it looks like to do that.
- 9:45
This cost per capability per second trade-off is actually really intense. Um, Citadel came out with this memo, um, a while ago, maybe two weeks ago. I loved it. The idea is that for the economy at large, simpler models might be the most cost-effective productivity-augmenting pathway.
- 10:04
They talk about this bifurcation on frontier versus everyday usage. I really believe that. And for every product, the definition of frontier versus everyday, the definition of saturated capabilities or model capability overhangs depends on your expertise on your product.
- 10:21
No one can replace that. And not all traffic is equal. It is a huge miss to send all of these to the latest Opus model. Some of these, absolutely.
- 10:31
Large-scale data analysis, when you do it on Notion, we'll recommend Opus, right?
- 10:36
When you triage an email inbox, if we're charging you to do that on Opus, we're ripping you off and ourselves. Think about where your traffic patterns are. And then think about how frontier lab model providers are structured today.
- 10:52
I mean, it's functionally an oligopoly, right? And that's fine because they're racing to the top, and I think the top is really hard and really important. This is not to say that products don't have a place for frontier difficult tasks.
- 11:04
I want everyone to nod and understand that's not what this talk is about. Understand when you need those tasks, and it's not everything. The problem with those tasks are is keep in mind how pricing is incentivized.
- 11:16
You can figure out who these players are. Either you are the best model. Everything above what AI can't do today is your market. You can basically price it as high as you kind of want.
- 11:28
If you're slightly behind that best model, all you need to be is like a dollar per million tokens cheaper, and you have the rest of the market. You know that economic theory about gas stations where the best gas stations are the ones that are right next to each other because they cover east and west the most?
- 11:43
Yeah. It's the same with model pricing, which means that price does not correlate with capability growth.
- 11:53
So for this complex task, understand what capabilities you need, but be the expert on what complexity is.
- 12:01
And keep in mind that who handles complexity changes. Um, oftentimes you'll see applied AI companies really be super outspoken on marketing with a specific lab. That's always kind of a red flag for me when they're not model agnostic, 'cause if you look at this graph, it basically shows that they're behind every month, right?
- 12:20
The new model and the new model provider of the best frontier capabilities change. And if you hitch your ride with one particular provider in exchange for, for instance, a larger discount, um, you're doing a disservice to your customers like half of the time, right?
- 12:37
So really think about if that discount is worth not actually having a frontier product.
- 12:43
And remember that that optionality is your leverage. If you don't have the capability to walk at any point, you are stuck. And again, I think that's probably the most expensive decision you'll make regardless of what discount you get or the engineering work to have model interoperability.
- 13:01
One option to navigate this is stay model agnostic. Have different models and capabilities in your system so that at any point, if pricing seems unfair or untenable, you are not out of business.
- 13:16
Notion's auto model does this really well. We have state-of-the-art models available always, um, but we also have an auto model there at the top that handles about seventy-five percent of our traffic, right?
- 13:29
We have the ability to switch between models in our product, and we also offer it to our customers so that they have access to these models without vendor lock-in.
- 13:37
That's part of our AI Switzerland approach. You guys love taking photos of slides. This is the slide. Okay. [laughs]
- 13:46
Model agnostic playbook. This is how you do it. Build for multimodal. It is hard to kill the cache and switch models mid-transcript. I, I understand that. We invest in that technology.
- 13:56
It doesn't even have to be per thread. Just think about your harness as model interoperability. Think about the cost per capability per second, not just the tokens. Here's a great example.
- 14:07
We posted this review when we, um, announced our partnership with Parallel as our web search provider. If you were to look at just latency of a single call or just cost, Parallel might not be the cheapest.
- 14:19
But if you have expertise in entire web search trajectories, you'll see how it differs. The granularity of this eval is what lets us make the best decisions for our customers because we understand all of the trade-offs on entire trajectories, not just single calls.
- 14:35
Switch fast and often, I think we talked about that, and give them something back. That expertise on use cases is also very valuable to frontier labs. We find that our evals and our early access program partnerships actually help us a lot with frontier labs and is something that we can exchange instead of extraordinarily large commits, and I
- 14:54
don't think the discount is ever worth the loss in optionality. That's a perspective you can choose to keep or not. The second option is moderate tasks, understanding open weights' place there.
- 15:06
Um, open-weight models are really strong enough to handle these tasks, and the possibility to RL on, on top of them has also kind of expanded the upmarket growth that they can cover.
- 15:17
I view open-weight models as basically lowering the barrier to entry on cost for our customers, and they also give you negotiation leverage. So it's kind of a credible alternative that's putting that downward pressure on pricing that if there's an oligopoly of two or three providers at the top is unavailable right now otherwise.
- 15:35
I think Kimi two six was probably the first time that we really saw a model that outperformed five two, GPT five two. GLM five two now is another five two.
- 15:45
Bombshell in the [REDACTED:location], that also probably does best here. But it's no longer the case where open-weight models are good for just SFT on small tasks. Really think about without RL if they're capable enough for what you need.
- 16:00
Um, and again, don't just think about external benchmarks. Be able to have expertise on your system. What are your tool errors? What's the actual latency that you need, right?
- 16:13
Here's an example of a benchmark that we posted. It's a little bit stale on purpose, right? But you get the idea.
- 16:21
Philip at Base10 showed this, showed this slide once, and I've stolen it ever since. Um- Great slide. Thank you. Are you here, buddy? Okay. We'll chat. Hi. [laughs] Um, well, he could come up and say it better, but the idea is that you don't have to be at the top, right?
- 16:37
I'm not trying to make a case that open weight is the best model out there. Um, the case being made, however, is that, um, the gap gets covered eventually.
- 16:48
So if the tasks that you're having today are good enough, then in six months they're probably covered by open weight. So be prepared now.
- 16:57
And the last thing is CPUs over GPUs. Um, we've, we've recently launched something at Notion called Workers. I don't think that the GPU is necessary for every job.
- 17:07
A lot of the jobs that we have are actually serving, um, discrete pieces of code. Like, you don't need an LLM to turn a CSV into a PDF. You don't need an LLM to talk to Notion tool calls if we have a CLI.
- 17:19
You definitely don't need an L- LLM to do deterministic SQL queries. This is where people become token poor very quick.
- 17:28
And I think the last option here, besides open-weight CPUs and optionality, is actually governance. Um, there's a lot of AI governance. Um, one is visibility, um, understanding who's using the data, understanding its maintainability and control.
- 17:44
When you have model optionality, you can offer a lot more to your customers.
- 17:49
Um, here's an example of how that governance works in Notion.
- 17:53
So final tips again, think about architecture, think about open weight, and build value that transcends tokens. So we're gonna depart Token Town. I know I said welcome to Token Town.
- 18:04
We're gonna spend the next ten minutes really thinking about what to do next. So I think the challenge of the next six months doesn't have to do with capabilities.
- 18:13
I think it has to do with security. Let's start there. There's this concept called the lethal trifecta. Simon Willison, I think, crafted this. If you have access to private data, exposure to untrusted content, whether it be through ingestion, MCP, email, right?
- 18:29
And the ability to ex- to communicate externally, and that can include, like, payloads in a web search. The second you have that system, you're exposing risk, and in fact, the more autonomous your system is, the more unsupervised this risk is.
- 18:42
I think that this is what builds valuable product, not just capability.
- 18:48
Same with sandboxes and computers. We talked about this, but it really is something that builds better determinism in your product and also better token economics for your customers.
- 19:00
In multi-agent orchestration, understanding what agents see and do and what persists, I think persistence of enterprise knowledge is something that's actually really not discussed enough. It's starting to be with some recent launches.
- 19:17
You know-- Oh, there is audio. [chuckles] So don't have your workflows look like this. And I think this is where most software factories are today, right? It's like, actually, your entire engineering time just spends time babysitting the factory, right? [chuckles]
- 19:30
I mean, I get it. Ours started off like this. Agent orchestration is one of the most difficult tasks of making factories work.
- 19:38
So okay. This is me telling T-Pain to tell people to buy Notion AI. And the reason I included this slide is that I'm gonna sell Notion for a second.
- 19:45
It's my job. Always be closing, always be selling, always be hiring. Come find me. But I'm gonna talk for a second about how Notion does this.
- 19:54
Today, we already have the ability to inspect tasks, and you can i-imagine any task that you look at, um, in a Notion document. You can have Claude actually go ahead and scope out what you need.
- 20:06
Uh, we've launched this Manage Agent capability today. So if I go ahead to the top of this task, I can actually ask Claude Agent to scope out the task, right?
- 20:18
Ideally, it's working. [chuckles] Um, and you'll see it'll actually populate, um, an entire spec of what needs to be done. In this example, it's not ready. It's gonna ask me a question.
- 20:30
Keep in mind this isn't a markdown file. This is an active document. Um, let's say I don't actually know the question, and I go ahead, and I ask my team, um, what to do.
- 20:40
Imagine that you can kind of tag in your team into these systems. MJ's our PM.
- 20:50
So in this example, she doesn't know. Usually, she does. But multi-agent orchestration is important. Maybe Claude Code isn't the best at customer voice, but Decagon is, right?
- 21:01
You can ask Decagon agents, we're proud partners with them as well, to collect the right data that you need. Okay. In this example, we think we know enough. We're gonna go ahead and actually, um, iterate through some of this flow.
- 21:17
I'm gonna skip ahead a little bit. We asked our TL what we needed. He replied. Again, it's a collaborative file and not just a markdown. And we can have Claude actually go ahead and spin up the PR.
- 21:28
Hopefully, this is looking a little familiar now. This is kind of the vision of Software Factories. It's what we're trying to host. Okay. Claude put up a PR. Maybe that's not enough.
- 21:37
Um, maybe I wanna go ahead and ask Codex what it thinks.
- 21:47
Great. Found two issues. You can think about this scaling in an actual factory. So today in Notion, you're actually able to orchestrate these agents together, and you're not committing to a lab.
- 22:01
You're committing to the concept that AI is augmenting and automating what you do.
- 22:08
This is real. I asked Rajiv if I could post this. [chuckles] This is how it works today internally at Notion. Almost all of our polish and large feedback like this is actually coordinated, um, through our Software Factories, both in terms of routing to the right teams and also having coding agents take the first stab.
- 22:26
Vercel does this as well, from staging to shipping to closing.
- 22:33
And we see massive ROI gains from our customers.
- 22:37
That's over three minutes saved on a given task. Imagine that at scale. So I think we're trying our hardest to think about the factory lens. We cannot do this without optionality, and we cannot do this without conviction that we understand what models are required for which tasks.
- 22:53
It's really wild out there, you guys. [chuckles] I get it. The market is really young. It's exceptionally opaque. It's moving fast. I'm super grateful for communities like AI Engineer to bring us together and, like, talk openly about these things and how we navigate it.
- 23:08
Um, I think we owe it to all of our customers to get it right and to be critical thinkers about how we navigate this together. I'm chronically online, unfortunately.
- 23:18
Um, you can always DM me on Twitter. You can email me. You can find me after this. Um, but thank you for yapping with me and thinking about this problem, and have a good day. [audience applauding] [upbeat music]