AI Engineer Code 2025
Stuffing Context is not Memory, Updating Weights is
Read the talk
Stuffing Context Is Not Memory, Updating Weights Is
A model can read your documents without learning to use them. Jack Morris explores what it takes to turn private knowledge into lasting capabilities—and where retrieval still belongs.
From a talk by Jack Morris
Before you start: Familiarity with language-model prompts, embeddings, and basic neural-network training will help; the article explains the adaptation methods as they arise.
Why can’t ChatGPT learn the thing you need?
ChatGPT can help prepare a presentation or cook dinner, yet it cannot explain why Jack Morris’s speaker pass failed at the conference entrance. Asked whether the Blue Jays won the World Series, it needs web search to know an outcome beyond its training cutoff. Asked to optimize an AMD GPU kernel, it encounters a different problem: the task is difficult, relevant examples are scarce, and those examples occupy only a small part of its training distribution. Missing knowledge is not one problem. It can be recent, private, or poorly represented in training.
Long-tail tasks expose that last gap. Asking a deployed model to practice does not itself give it a training loop. Nor can a general model infer the terms of your BlackRock partnership, which shirt you would prefer, how your company monorepo works, or how you write email. Patient histories, opposing counsel’s arguments in the Martinez negotiations, and answers buried in an internal wiki all require information the model may never have encountered.
There are three broad ways to make that information available:
| Approach | Where the knowledge goes | What happens for a question |
|---|---|---|
| Full context | The input window | Supply the relevant corpus directly |
| Retrieval-augmented generation | An external store | Retrieve useful material into context |
| Learning in weights | Trainable model parameters | Answer using an adapted model |
The first two change what the model can read during a request. The third changes the model itself. Morris’s central question is how to make that third option useful enough to become part of ordinary application development.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reading everything has a cost
For a bounded collection, copying everything into context is an excellent starting point. A small company’s records, a history of roughly a hundred World Series, or one patient’s medical record may fit well enough for the model to answer useful questions. The difficulty appears as the input grows: every request carries more information through the inference system, increasing both cost and waiting time.
Morris recalls that pasting his roughly eighty-page thesis into Claude made the interaction feel about ten times slower. His throughput slide gives a more specific comparison: Llama 8B, peak throughput on one H100, excluding prefill.
| Context tokens per user | Output tokens per second |
|---|---|
| 1,000 | 10,000 |
| 128,000 | 130 |
These are serving-throughput figures under the slide’s conditions, not the speed of a single Claude response. They illustrate how much longer context can change the economics of generation even before counting prefill.
The architectural explanation starts with self-attention. In conventional full attention, each input token can interact with every other input token. Four tokens give a 4 × 4 attention matrix with sixteen entries; twelve give a 12 × 12 matrix with 144 entries.
That all-pairs structure creates a quadratic constraint when processing a sequence. Morris emphasizes the resulting memory pressure: making the context window larger does not remove the work required to use it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Accepting tokens is different from using them
Large advertised windows seem to challenge that limit. Morris cites Grok 4 at two million tokens and Gemini 3 at one million in the talk’s contemporary comparison. But accepting an input without crashing is different from reasoning across it. A model can remain operational while becoming less reliable at finding and combining the information that matters.
Chroma’s Context Rot report examines this distinction by increasing input length, including settings where relevant information stays fixed while other material grows. Morris highlights Claude’s comparatively strong showing in the displayed graph, which he feels better matches users’ experience than some standard benchmarks. He then points to severe task degradation around 10,000 tokens in the illustrated irrelevant-context setting. That is his reading of this particular comparison, not a universal failure threshold. The outputs can still be grammatical while failing to solve the problem—a pattern he also connects to complaints about Claude Code as its context fills.
More efficient architectures address part of the problem. Mamba, other state-space models, linear attention, hybrid attention, sparse attention, and sliding windows all offer different ways to reduce computation or memory use. The remaining question is whether they preserve enough information and computation to improve answers as more material arrives. Efficiency alone does not establish that capability.
The example is MiniMax-M2, which retained conventional full attention after exploring alternatives. MiniMax’s explanation describes a production trade-off involving quality, evaluation, and infrastructure. Morris uses that choice to separate two obstacles: fitting an enormous input into the system, and reasoning effectively over that input. Industrial collections can extend to billions or trillions of tokens, far beyond the scale at which either problem is settled. When he asks who works with RAG weekly, he estimates that more than half the room raises a hand.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Embeddings are convenient, but they are not encryption
RAG has a practical advantage: its infrastructure already exists. Morris names turbopuffer, Weaviate, Chroma, and Mongo among the available vector-database options, with different price, speed, and storage trade-offs, including S3-backed offerings. He describes internal question answering and ChatGPT memory as embedding-backed systems, invoking Karpathy’s analogy of embeddings as an LLM’s file system. He dates that assessment to November 22, 2025. His prediction is that embeddings will not remain the only way to provide persistent knowledge.
The appeal is also developer experience. Producing an embedding can take roughly five lines of code; building a reliable training workflow is substantially harder. Yet when Morris asks whether anyone is completely satisfied with their RAG system, nobody volunteers in the exchange. The ease of constructing the pipeline does not settle the quality of the resulting memory.
The basic abstraction is one vector per document. On the slide, those vectors are projected into two dimensions so each document becomes a dot. The actual stored representation is a list of numbers that a person cannot read. That unreadability can be misleading: another learned system can map the vector back toward the original text. The displayed example includes a sentence about racehorse Mage winning the 2023 Kentucky Derby, its vector, and a reconstruction with altered details.
The reconstruction becomes stronger through iteration: generate a candidate text, compare its embedding with the target, and correct the candidate over multiple rounds. In Text Embeddings Reveal (Almost) As Much As Text, Morris and collaborators studied this inversion process. Morris summarizes the result as roughly 90% exact recovery at a particular text length; the paper’s reported setting is 92% exact reconstruction of 32-token inputs using iterative correction and re-embedding, with experiments on two embedding models. This does not imply the same recovery rate for arbitrary documents or embedding services.
An embedding should not be treated as a privacy boundary merely because it looks like numbers. Morris raises the issue in the context of sending embeddings to external stores, naming turbopuffer and Pinecone. The security consequence is that recoverable text can remain present in a representation even when the original document is not sent alongside it. He also flags the operational difficulty of running vector infrastructure at scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When Visa and Mastercard are too similar
A second problem is that a general embedding model supplies a general notion of similarity. Morris describes a corpus split approximately evenly between Visa and Mastercard documents. In a space representing all possible subjects, both brands belong to a small region about credit cards. But in this application, the brand distinction is essential: a Visa query should never retrieve Mastercard material. Semantic proximity works against the application’s requirement.
The contextual approach changes how a document is embedded. A two-stage model incorporates surrounding-document information while producing the target vector. It can therefore represent a Visa document with the knowledge that the collection consists largely of Visa and Mastercard documents, making the distinction between them more salient. Amex appears in the presentation’s illustration, but Morris clarifies that it was not in the experiment. In the shown contextual example, Morris reports a Visa–Mastercard similarity of 0.144, with higher similarity among Visa-containing material.
Morris reports improvements on specialized datasets involving climate, arguments, financial questions, and scientific articles. When asked about adoption, he says contextual embedding methods are used inside OpenAI, Anthropic, and other companies. The implementation requires extra machinery: obtain surrounding-document embeddings, then condition the target representation on them. He describes stronger benefits on niche collections and little improvement on the broadly distributed MS MARCO task, where a global semantic space already fits the problem better.
Better data remains his first recommendation for improving an embedding system. Contextualization helps adapt similarity, but it does not make a fixed-dimensional vector capable of representing every possible relationship a future query might require. Morris points to a combinatorial limitation: the space of possible text relationships can exceed what a fixed representation can preserve for retrieval.
The practical failure cases he wants to capture involve associations across documents and answers implied by a collection rather than explicitly stated in one passage. He also acknowledges how difficult it is to build convincing benchmarks that ordinary retrieval cannot solve. That admission matters: the proposed target is richer reasoning over a corpus, but identifying tasks that cleanly measure the missing capability remains part of the research.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pay for reasoning at training time or inference time
An audience question introduces agentic search: a model makes successive queries, inspects what it finds, and then answers. Morris distinguishes this from the ordinary RAG pipeline he has been criticizing. In principle, an agent could read a whole collection and reason over its relationships. The obstacle would be the expense of doing that work.
Deep research is a promising example of spending more at inference time. The audience describes a system that searches many sources but retains only a subset in context. Morris welcomes the direction: taking time, searching repeatedly, and thinking through the results are useful ways to buy better answers. His weight-learning proposal spends more effort earlier, so the deployed model already knows the collection.
| Strategy | Main place to spend extra work | Intended benefit |
|---|---|---|
| Deep research | Searches and reasoning per request | Investigate information as needed |
| Weight adaptation | Data generation and training upfront | Reuse learned knowledge during inference |
Better results can require more investment in data, training, or inference. RAG is attractive partly because a basic version is inexpensive to build and operate.
Another audience member revisits Visa and Mastercard and asks whether a knowledge graph could augment the system. Morris defers that discussion rather than assessing the approach. The next part of the talk therefore explores parameter updates without establishing that they supersede graph-based alternatives.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What should a model spend its capacity on?
Suppose you have collected a small dataset from your own work. Putting it in context may already help, but some questions can remain difficult even with all the source material present. The weight-learning goal is to make the model use that knowledge more effectively while preserving useful capabilities it already has.
Model capacity is finite. In How much do language models memorize?, Morris and collaborators estimate approximately 3.6 bits per parameter for the tested GPT-style models, separating memorization from generalization. At that rate, one billion parameters corresponds to 450 million bytes, or 450 MB of information. This corrects the talk’s spoken conversion; the estimate is not a universal storage specification for ChatGPT. A model allocates its finite capacity to fitting its training distribution and fails to retain everything else.
Morris illustrates the allocation problem by asking Claude for the capital of the smallest province in Tajikistan. It supplies a detailed answer without search. That is impressive, but it may be irrelevant to a company documentation assistant. Ideally, the assistant could replace irrelevant knowledge with the organization’s own information. Precisely locating, deleting, and replacing knowledge in weights is not yet a solved operation. The slide’s phrase neural file systems names this ambition; Morris uses the simpler term weights in the talk.
Three design choices follow:
- Learning objective: Should adaptation use supervised fine-tuning, reinforcement learning, or another objective?
- Training material: Should the model read the raw corpus, or a larger set of generated examples derived from it?
- Architecture: Which parameters should change when personal models need repeated updates?
The source collection might contain Mastercard documents, personal information, or coding traces gathered across November and December. Whatever the source, Morris expects learning beyond a basic RAG system to require meaningful GPU expenditure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A model memorizes 3M—and fails to write a poem
The simplest experiment is to train directly on the documents using next-token prediction. Morris’s team tries this with a 3M 10-K financial report: a structured document whose contents interrelate in ways they want the model to understand. Morris reports zero training loss on the report, even without updating the entire model. By that measure, the model has memorized the document extremely well.
The next test asks for behavior that is not copied directly from the report: write a poem about 3M in fiscal year 2025. The response is, “The passage of a passage is a poem.” Then it stops. Memorizing the training sequence did not produce useful command of its subject.
Morris diagnoses two problems. First, raw document text is not necessarily the right training distribution for the questions and tasks the assistant will face. Second, the update can make the model excessively specialized, damaging its general behavior and encouraging document-like continuations. Improving this experiment requires changing both the learning material and how the model is allowed to change.
Karpathy’s experiment teaching a small model about himself suggests a different route. Instead of repeatedly training on a single factual example, he generates a diverse set of task examples and fine-tunes on those. The variety teaches the desired behavior through multiple formulations. Morris describes it as a successful instance of teaching novel behavior through synthetic data.
Synthetic continued pretraining applies a related idea to a small source corpus. Its EntiGraph method extracts entities and generates diverse connections among them. The resulting material is meant to preserve the source facts while expanding the ways those facts are expressed and related. A small collection can thus support a much larger training distribution without first collecting a correspondingly larger set of original documents.
In the displayed comparison, direct fine-tuning drops below the model’s starting performance: reproducing the source more faithfully makes it less able to answer differently phrased questions. Synthetic expansion does better. Morris reports that synthetic-data training surpasses GPT-4 on the paper’s dataset at a loosely described scale of roughly 100 million to nearly one billion tokens; he does not specify the exact crossover or baseline configuration in the explanation. The useful mechanism is the expanded range of learning examples, not simply more repetitions of the original text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate material the model can learn from
Synthetic expansion can take several forms:
- Active reading: Ask the model what kinds of learning material should be generated, then generate them.
- Cartridges self-study: Use question answering and self-quizzing to turn source material into practice.
- Rephrasing the web: Produce alternative expressions of pretraining material at large scale.
These approaches vary the learning experience rather than relying exclusively on the original document’s wording and order.
Morris also points to Datology’s synthetic-data work and to SEAL, or Self-Adapting Language Models, which asks a model what data it should generate to improve itself. He describes SEAL as working in constrained scenarios. The interesting result is that a model can sometimes help design its own adaptation data, not that it can improve without limit. The broader opportunity is to turn a small factual source into a large, useful collection of learning examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Change a small part of the model
Better data still leaves the problem of catastrophic forgetting. A classic example is a model that learns handwritten digits, then learns house numbers and loses its ability to recognize the handwritten digits. Updating a language model on a new collection can similarly interfere with old capabilities. Morris presents several ways to restrict the update, while explicitly treating the choice among them as unsettled.
| Method | What changes |
|---|---|
| Full fine-tuning | All model parameters |
| Prefix tuning | Learned prefix key/value state |
| Added expert | An optionally routed MLP component |
| Memory layer | A differentiable lookup table |
| LoRA | Small low-rank matrices adapting linear layers |
Morris describes prefix tuning as training the KV cache. An added expert expands what an MLP can route to; a memory layer offers a different, table-based place to store learned information. LoRA, short for low-rank adaptation, constrains a linear-layer update through smaller matrices. His illustrative scale is a ten-billion-parameter base model controlled by ten million trained parameters.
A minimal PyTorch layer makes the LoRA restriction concrete. For a frozen weight matrix W, the learned update is B @ A; the rank determines the size of that update’s trainable representation.
python
import torch
from torch import nn
class LoRALinear(nn.Module):
def __init__(self, base: nn.Linear, rank: int, alpha: float):
super().__init__()
self.base = base.requires_grad_(False)
self.scale = alpha / rank
self.a = nn.Parameter(base.weight.new_empty(rank, base.in_features))
self.b = nn.Parameter(base.weight.new_zeros(base.out_features, rank))
nn.init.normal_(self.a, std=0.01)
def forward(self, x: torch.Tensor) -> torch.Tensor:
update = (x @ self.a.T) @ self.b.T
return self.base(x) + self.scale * update
layer = LoRALinear(nn.Linear(8, 4), rank=2, alpha=2.0)
optimizer = torch.optim.AdamW(
[parameter for parameter in layer.parameters() if parameter.requires_grad],
lr=1e-3,
)
The base weights remain frozen, and b starts at zero, so the initial layer output matches the base layer. Training can change a and b without directly overwriting W. This illustrates the common idea behind Morris’s comparison: attach a relatively small trainable component to a capable model and preserve the original parameters. It reduces the scope of the update; it does not guarantee unchanged behavior.
For personal models, the desired properties are small stored changes, resistance to forgetting, enough capacity to capture useful facts, and fast inference. A separate full copy of Kimi for every user would be impractical at the trillion-parameter scale Morris invokes; exact storage also depends on numerical precision. Thinking Machines’ Tinker training API represents the shared-base alternative: independently train user-specific LoRAs and batch their use over an unchanged base model. Morris’s description of this as having essentially no cost refers to sharing the base computation and infrastructure, not to a measured zero-cost training or serving service.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How much capacity does adaptation need?
LoRA Learns Less and Forgets Less compares low-rank adaptation with full fine-tuning in programming and mathematics. Morris highlights its supervised-learning trade-off: the smaller update learns less of the new task, but also causes less degradation of existing capabilities. Restricting the update can preserve knowledge precisely because it limits how much the model can change.
Thinking Machines’ LoRA Without Regret supplies a contrasting result: LoRA can approximately match full fine-tuning in the reinforcement-learning settings Morris discusses. The displayed plots use Llama-3.1-8B on GSM and MATH, comparing full fine-tuning with LoRA ranks 1, 16, and 256 across learning rates. Morris interprets the contrast as evidence that RL can require smaller changes than supervised fine-tuning. The report also finds that LoRA can match full fine-tuning on small-to-medium supervised datasets; adapter capacity and dataset demands matter, so SFT does not categorically require updating every weight.
Morris then presents preliminary TinyLoRA research, still unreleased at the time, extending the idea behind LoRA-XS. Morris reports approximately 91% GSM8K accuracy from reinforcement learning with fourteen trainable parameters. He also raises a concern about the Qwen model’s training data, which limits how strongly he interprets the result as newly acquired reasoning.
An audience member asks about the smallest possible update. Morris says the one-parameter experiment improves performance by “five percent,” without specifying whether that means a relative increase or percentage points. The mechanism is more expressive than changing a single isolated weight: one scalar controls many random projections, producing changes throughout the model while leaving only one number to train and store.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prefixes, memory layers, and sparse learning signals
The evidence does not identify a single best parameterization. Morris cites a paper favoring prefix tuning over LoRA, then recalls Meta colleagues who preferred LoRA. Systems support could influence the eventual choice: prefix tuning works with KV-cache infrastructure that serving systems already manage extensively. At the time of the talk, he describes concurrent multi-LoRA training kernels in open source as a missing piece, contrasting that difficulty with the infrastructure Thinking Machines is building around LoRA.
Memory layers offer another route. Think of an expert implemented as a large differentiable lookup table. In work Morris attributes to Justin Lan, training updates only specified small portions of that table. On the displayed learning-versus-forgetting plot, moving right means learning more, while moving down means forgetting more. Morris describes the memory-layer method as learning nearly as much while exhibiting almost no forgetting in that study. He finds it promising for preserving a base model, but the conflicting results across methods leave the architecture question open.
The audience returns to the TinyLoRA experiment and asks whether supervised fine-tuning came first. Morris estimates that SFT needed roughly 1,000 times more trainable parameters for equivalent performance in these experiments. The revisited slide identifies Qwen2.5-7B-Instruct, GRPO, GSM8K, and update size in parameters, with separate markers for TinyLoRA, LoRA-XS, and LoRA.
His proposed explanation concerns the amount of information in the learning signal. SFT supplies tokenwise cross-entropy targets, potentially including every token in a reasoning trace. In his simplified account, RL supplies a sparse correct-or-incorrect reward instead. That can steer behavior already available in the base model without requiring an equally large store of new information. The experiments compare GRPO and SFT as separate alternatives; they do not run GRPO after an SFT stage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Training and retrieval will coexist
A practical question changes the emphasis: what if there are only a few documents, a fact appears once, or the data changes frequently? Morris expects both context and RAG to remain useful. His illustrative deployment performs a large training update periodically—perhaps daily—and still retrieves information on every served request. He offers no measured document-count or economic crossover at which training becomes preferable. Corpus size, information frequency, and update rate remain variables that need analysis.
The proposed adaptation targets assistant and agent completion models, not primarily embedding models. Morris sees more potential in teaching the answering model than in further refining retrieval, but his suggested order-of-magnitude gains are forecasts rather than measurements. Asked what remains difficult even with excellent retrieval, he points to chunking: retrieving some relevant passages may still omit the evidence and relationships needed to reason across the collection. He argues that some datasets resist any chunking scheme that supplies everything a question needs.
At consumer scale, the audience asks about personalization for ten million to one hundred million users. Morris suggests that a few megabytes of LoRA state per user could be manageable, comparing that storage burden with video services. Continual updates are harder than storing adapters; once-daily training is his near-term example. He explicitly concedes that the overall approach is more expensive than current systems.
Audience members point out that the other approaches can improve too: compress context, compress material before retrieval, and combine RAG with SQL and knowledge graphs. Morris agrees that progress will continue along all three axes. When pressed for a decision boundary, he offers a qualitative distinction: a large corpus that the base model has never learned is a promising target for adaptation; small, frequent additions are better candidates for inference-time investigation closer to deep research.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From proprietary documents to a shorter prompt
How would synthetic generation help a company with millions of proprietary documents and its own terminology? Morris starts with information density. Many documents may repeat the same formatting and contribute only a small amount of new information. The useful transformation would identify what each document adds, then express that knowledge in examples the model can learn from.
His proposed starting procedure is:
- Identify the new information in each document rather than reproducing its formatting.
- Generate question–answer pairs that expose that information.
- Shape the question distribution around what users will actually ask.
He calls this a research direction, not a general recipe. Question answering gets part of the way toward the desired training distribution; covering every useful question and relationship is the harder goal.
Returning to the 3M example, another question asks what the inference prompt would contain after successful adaptation. In the ideal case, only the user’s question: no repeated financial report and no extra system prompt supplying its contents. If the adaptation concerns a single document, even the document identity might be implicit. That interface depends on the model having learned the information successfully and on the information remaining unchanged.
An audience member challenges the premise that weights are the best place to store information. Morris explicitly narrows the claim: weight storage is an underused capability, not necessarily the best medium for everything. A repository with millions of tokens makes repeated reading expensive. Compressing the behavior induced by a long prompt into an adapted model could reduce subsequent inference costs. Whether synthetic training can also exceed the performance of having the source in context remains a research question; lower recurring inference cost does not by itself establish lower total cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What happens when the training data is wrong?
Generating training material from user inputs creates an adversarial surface. A malicious user could supply information designed to sabotage the resulting adaptation. Morris says he has not previously considered the question and offers no mitigation. He distinguishes damage contained within a user’s personal model from contamination introduced when information is aggregated across users. Deliberately negative feedback provides another example; an audience member also raises cultural differences in feedback behavior, without identifying the study.
Versioning presents a less exotic but equally consequential problem. A company changes a one-line policy from honoring something to no longer honoring it. Should training restart from the base model, or update the existing adaptation? How does the system prevent the old policy from returning in answers? Morris speculates that a good synthetic-data strategy might resolve conflicts during dataset construction or omit obsolete documents, but he does not have an established method.
A follow-up asks how proprietary training data should override contradictory pretraining knowledge. Morris’s answer is that changing the storage mechanism does not eliminate the authority problem. If outdated information appears repeatedly, a naive retriever can keep returning it and a naive training pipeline can keep reinforcing it. Learning in weights still requires deciding which information is current and authoritative. The conflict-resolution work remains.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialized models that can keep learning
Parameter-efficient updates could also reopen possibilities for federated learning. Sharing gradients across machines becomes expensive when the model is enormous. Morris suggests that exchanging updates for a small trainable component could reduce that burden, especially in RL settings where substantial local work occurs between gradient updates. He has not conducted the proposed federated research; he presents it as an opportunity created by smaller updates.
The next question contrasts specialized knowledge with the idea of a small, largely knowledge-free reasoning engine. A lawyer does not memorize the whole legal code; the lawyer knows how to find and use the relevant material. Perhaps a compact model should similarly learn where to look, especially for information that becomes stale. Morris agrees that much memorized knowledge is unnecessary for a particular application, returning to the Tajikistan example. But separating reasoning from all factual knowledge is difficult. His preference is for models that are strong in the domains their users care about and weaker elsewhere, while still using tools when needed.
The final technical question concerns time. Given information from day one, day two, and day three, should the system concatenate the data, train sequentially, or train separate adaptations and merge them? Morris has no preferred method or reported result. These choices expose the remaining gap between a model that can be fine-tuned once and a model that can reliably keep learning.
Morris closes by describing a new company working on that broader problem and inviting interested users and San Francisco engineering referrals. Its scope is not only temporal information. The goal is to build models that people can teach: models that acquire useful knowledge from their work, preserve the capabilities that still matter, and make that learning available in later interactions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Chroma's experiments on how increasing input length changes language-model performance.
MiniMax explains the quality and engineering trade-offs behind M2's attention architecture.
Research demonstrating iterative reconstruction of text from embeddings, including 92% exact recovery for 32-token inputs.
Measures memorization separately from generalization and estimates capacity in tested GPT-style models.
EntiGraph expands source documents into diverse synthetic learning material through entity relationships.
Compares learning and forgetting under LoRA and full fine-tuning for programming and mathematics.
Experiments examining when LoRA matches full fine-tuning under supervised learning and reinforcement learning.
Further reading
Training and evaluation code for embeddings that incorporate surrounding document context.
Updates since the talk
The February 2026 TinyLoRA paper reports GSM8K reasoning adaptation with just 13 trained parameters.
Read the complete timestamped transcript
- 0:00
[on-hold music] Let's talk about ChatGPT.
- 0:22
I think, like, ChatGPT knows a lot of things. It's actually extremely impressive. I use it all the time. I used it to help prepare for this presentation. I used it to cook last night.
- 0:32
Um, uh, you know, very-- Like, growing increasingly dependent, and yet there's a lot that ChatGPT doesn't know. Like, um, it didn't know why my speaker pass wasn't working when I was trying to get into the building and it, uh...
- 0:46
If you ask it, "Did the Blue Jays win the World Series?" the answer is no. And I know that because I watched the World Series. But ChatGPT doesn't know that if you don't enable web search because it has something called a knowledge cutoff.
- 0:55
So all the training data is kind of segmented by date, and things after a certain date are not known by ChatGPT, like, unilaterally. Uh, if you ask ChatGPT, "Help me optimize this kernel I wrote for AMD GPUs," it's so bad at it, and I think there's a few reasons for this.
- 1:13
One, it's really hard. Two, uh, there's not a lot of data for it. But three, I think it's more that the data that does exist is such a small portion of its training data that it just, like, can't do it very well.
- 1:25
And so a lot of tasks like this, which I, I would guess a lot of you face in your jobs, like, the things that are more niche or, or are called long tail, are really hard for ChatGPT to do.
- 1:35
Even if you say, "Please," like, "P-please," or like, "I want you to learn more about this or practice." Like, it can't learn more about this. It can't practice. It, it doesn't know, uh, what to do when you ask it that.
- 1:47
And, uh, yeah, if you ask what are the terms of our partnership agreement for BlackRock, it doesn't know about your company. Which of these shirts should I order from Amazon?
- 1:55
Implement a new feature, uh, in our company monorepo. Write an email in my style. Diagnose this patient given their history. What arguments did the opposing counsel use in the Martinez settlement negotiations?
- 2:08
Uh, is this question already answered on our company internal wiki? Like, none of these things are possibly answered by ChatGPT because they're not in the training data, or they're too niche, or they require some data that's not available to it.
- 2:21
So I think, like, the question I wanna talk about today is, like, what's the right way to solve this problem? Like, if we wanna build new systems that actually know the things we want them to know, uh, how, how should we build them?
- 2:31
And I think, like, the way I wanna think about it is, like, how do we take some knowledge and inject it into the parameters of the model? Like, what's the right way to do this?
- 2:41
And, like, the way that I think about it, and I think the way this manifests in my research and other people's research, is there's three ways. There's full context.
- 2:49
You can take as much stuff as you can and cram it into the language model. There's RAG, or Retrieval-augmented generation, where you have so many things that you can't fit them all in, and so you retrieve the most useful ones and then feed them in.
- 3:05
And then there's this third thing, which I think is, like, really new and no one is doing it yet, which is training things into weights. And I wanna-- What I mostly wanna talk about today is, like, why I think we should be training things into weights.
- 3:16
But I'm gonna start with the other two. And also, I guess, like, along the way, about ten percent of the time, I'm gonna be shilling my own research, but I'm gonna, like, try to be honest about it, and you can just tune me out if you want.
- 3:28
So I think, like, the easiest way to solve these problems is to put everything into context. Like, if you work at a small company, or, um, all you care about is, like, maybe the one hundred World Series that have occurred, you can kind of copy all the data and paste it into ChatGPT or paste it into Groq
- 3:45
or whatever model you use, and that's finite enough that the model can understand.
- 3:51
And this, like, works, works pretty well. I think that this is something that got people really excited for a while a few years ago. I have this example of, like, a doctor answering a question from a medical record.
- 4:02
A medical record is small enough that it can presumably be, like, inputted into the context of the model, and the model can do pretty well. I think there's a few problems with this.
- 4:11
Maybe the main one is just that it's so expensive. Like, if you do anything like this in your day-to-day workflow, you put, like, a ton of tokens into context and start generating.
- 4:20
I mean, one, it's gonna cost a lot of money, like US dollars. But two, it's just so slow. Like, um,
- 4:28
you know, a few months ago, I was writing my thesis, and I wrote it myself, but I did ask for some feedback a few times from Claude. And, like, the second you paste in, I, I don't know, it's, like, maybe eighty pages of text or something, it-- like, as documents go, it's medium length.
- 4:46
I paste it into Claude. The second you paste it into Claude, everything slows down by 10X or something. I have this stat here that if you have one thousand tokens of context,
- 4:55
we can output ten thousand tokens per second. If you have a hundred and twenty-eight K per to-- a hundred and twenty-eight K tokens of context, we can output a hundred and thirty tokens per second.
- 5:03
So that's, like, several orders of magnitude of slowdown, and I think we've all faced this. So it's very annoying, and it's hard to imagine how we can get around this.
- 5:12
Um, I'll give you, like, the quick background from the research world, which maybe people know, which is this inherent limitation in the models we use. The models we use are transformers.
- 5:21
Transformers look like this. The real problem with transformers comes in this one little, uh, box right here called self-attention. The problem is that all of the words that go into the transformer need to look at each other, and this has a quadratic dependency.
- 5:36
So if there's four words, four tokens, maybe the matrix has sixteen entries. If there are twelve tokens, there are a hundred and forty-four entries. And we can manage this for a while, but at some point it becomes infeasible.
- 5:47
Like, especially from a memory perspective, we can't-
- 5:50
Can you hold the mic?
- 5:51
From a memory perspective, we can't keep all these things in context.
- 5:56
You might say, "Well, Jack, Groq four has two million token context window." Yeah, two million token context window. It's ex-it's a very large number. Gemini Three dropped, uh, during this conference.
- 6:08
Gemini Three has one million token context window. Uh, you also might ask, why did Gemini Three not do a larger context window even though it came after Groq? And I think the reason is because there's a difference between the model not breaking when you put in that many tokens and the model actually, like, properly reasoning across many
- 6:28
large chunks of tokens. And I think the second part we're still figuring out. I think people have realized how to train models that don't break with more and more tokens, but we haven't really gotten to the point where we can train models that truly work as well on a million tokens as they do on a thousand tokens.
- 6:47
And if you're more curious about this, there's this really good report from Chroma called Context, Context Rot, um, about how performance degrades when you add just, like, other stuff into the context.
- 6:59
So this graph shows, like, the larger the context grows, even with the same finite amount of relevant information, the LLMs get worse and worse. And I think, like, two things to observe here that I think are interesting.
- 7:11
One, Claude is the best by far. I like graphs like this because I feel like if you talk to people, a lot of people think Claude is the best.
- 7:17
But if you measure on a lot of standard benchmarks, it actually is worse. But then you use it, and you're like, "Oh, something's better here." So I like this because it captures what people actually say to me, but I also like it because at-- Once you get here, the performance is horrible.
- 7:30
So, like, if they, if they enter a bunch of relevant stuff that doesn't actually help you solve the problem, once you get to ten to the four tokens, which is ten thousand, like, the models don't work at all.
- 7:42
And even though they're not breaking, like they're outputting things that make sense and are grammatical, they're not actually solving the problem. So context rot is a huge issue. Um, maybe like just anecdotally, if you look up, there's a ton of people saying stuff like this.
- 7:58
Like, "Oh, what? The context window is so long. Why does it not actually work?" Or people think Claude Code, when it fills up the context window, sort of like stops working.
- 8:06
Um, there's a ton of people working on these efficient architectures that you might hear about, like, um, Mamba, state-space models, linear attention, uh, hybrid attention, sparse attention, sliding window.
- 8:17
They're all more efficient, but they basically have the same properties of transformers. Like, even if they can operate, uh, in a faster time or with a lower memory requirement, there's some trade-off in the terms of performance they give you.
- 8:30
So even if you build a linear attention model that can fit infinite context, it's not good. Like, it's not going to be able to solve the problem you have, which is: How do I actually, like, reason and get smarter when I input more tokens into the model?
- 8:47
There's so many examples of this. I saw this recent post. If you're like kinda deep in the model architecture world, maybe you've seen this. This is like a couple weeks ago.
- 8:56
There's a new [REDACTED:origin] model, Minimax M2, that's one of the state-of-the-art open models, and a bunch of the other [REDACTED:origin] labs have been pushing these new hybrid architectures that are like more efficient and can take longer context, and Minimax M2 just didn't do that.
- 9:10
They just used sort of like the regular quadratic attention that I was showing you, and they have this really long story about how they tried and tried, and it's basically just not worth it.
- 9:19
There's like an inherent trade-off in how much computation you use and, and how good the models are. And so even if you can technically build a model that doesn't break at millions of tokens, it's not actually better for any of the tasks they care about.
- 9:33
So no one is really doing this. And I think to conclude, we think that, like, we're pretty limited by the context window in full context. There's, like, one systems problem that you can't put millions of tokens into the model, and then there's another reasoning problem that even if you can, the models don't actually get better.
- 9:50
So it's probably not practical. And I think if, if you work in industry, I'm sure you see document sets that are much, much larger, like on the order of, I don't know, billions to trillions of tokens.
- 10:02
And even though we're getting better at training the models and the system side, we're getting much better at running them more efficiently, faster, cheaper, we're not near fitting trillions of tokens into a model.
- 10:14
I think, like, that's pretty far off. So I would guess a lot of you are doing RAG. How many people in this room use or work on a RAG system on like a weekly basis?
- 10:25
That's actually pretty crazy. Okay, so over half for sure. So now we're gonna talk about RAG. I'm gonna talk about why it's good, and then I'll talk about why I think, um, it's fundamentally limited and the products of the future will use something better than RAG.
- 10:44
So if you use RAG, you probably use a vector database. There are many vector databases. I think I know some of these. Turbo Buffer, Weaviate. Now they're on S3.
- 10:55
That's Chroma. I made this slide. Uh, Mongo. Uh, there, there are many different vector databases. They all offer you, like, slightly different trade-offs. They give you your vectors for cheaper, faster.
- 11:07
Um, vector databases are the way that memory works in production. If you're using a company internal question answering system, it's, it's definitely running on RAG, which is powered by a vector database which stores embeddings.
- 11:20
ChatGPT memory, uh, uses embeddings. Uh, Andrej Karpathy has this diagram from last year, two years ago actually, of what the o- an operating system that runs on language models would look like, and he called embeddings the file system of LLMs.
- 11:37
Um, I think that's true in today's terms. Like, today, November twenty-second, twenty twenty-five, probably, like, if you think of what you're working on as an operating system, the file system is embeddings.
- 11:48
But I think embeddings are the file system of today, and they're not the file system of the future, and that's what I'm gonna talk about today.
- 11:56
I, I also want to point out that they're extremely easy to use. Like many of the tools I'm going to talk about at the end of the talk That are like out related to training things into models are just fundamentally harder.
- 12:06
But this is just really nice, and we can all take a moment to appreciate it. You just sort of take your text, and then you, like, run this, and, and that's all.
- 12:15
It's like five lines of code. That's, that's really, really good. Um, the problem is they just aren't that good, and they have a lot of problems, I think. Um, which I think also...
- 12:26
Okay, how many people work on RAG or experience a RAG system and are satisfied completely with- [laughing] -like what you
- 12:37
get? Okay, that's great. So I think we're all kind of in agreement here that maybe there, there could be something more. Like, even if we don't know exactly what it is, there must be something else out there.
- 12:46
Um, I'll talk about a few problems that I've run into in my own research. So let's, like, start with this abstraction. So this is the vector database that powers RAG.
- 12:56
Every dot here is, is supposed to be a document. So the document goes through the LLM. The LLM is trained to give you just this one vector that represents the document.
- 13:06
I projected them down to two dimensions for the slide, but each doc- document is one dot. Um, if you actually look at what's in the vector database, it looks like this.
- 13:15
So lots of numbers. There's no one on the wor- in the world who can tell, tell you what this means. Um, one thing that I think is interesting is that even though they look random and no one can actually read them, if you build a system to read them, it works pretty well.
- 13:33
So, like, if you're working in, uh, RAG and you're sending someone embeddings, you're actually sending them something analogous to text. And I think this is important because a lot of the actual architectures like TurboPuffer, Pinecone, what have you, they store only embeddings.
- 13:49
And so, like, maybe there's this false premise that if you just send them embeddings, there's no security flaws. But actually, um, even slightly motivated person can build this system here, this white arrow on the right, which takes the embedding and produces maybe not the exact same text, but something extremely close to it.
- 14:06
This is what I worked on for, like, about a year of my PhD. This is a animation of, like, so I type in the sentence. It goes into the embedding model.
- 14:16
It gets stored in a vector database, and then we run this. It's like a multi-round correction thing. And then by the end, we actually can get most... I think our research has, at a certain length, we can get ninety percent of text back exactly from vector databases.
- 14:29
So the takeaway here is that there's no, uh, security benefits to using a vector database. And also, they're very hard to run at scale. So this is, like, an inherent problem for people with sensitive data.
- 14:41
That's the paper. Um, I think a second problem that I personally have with embeddings is that they're not adaptive. Like, there's this one universal sense of what the world looks like that's captured in these vectors, and it's not adjustable based on what you work on.
- 14:56
So, like, to give you a concrete example, we embedded a bunch of databases, or we created a database of a bunch of embeddings of credit card-related documents. I think we had half of them that were from Mastercard and half of them that were from Visa.
- 15:12
But if you actually look at where the embeddings get stored, um, I guess it's not in this picture, but it's, like, only right here. So even then there's this, like, really large space of kind of all possible semantics.
- 15:24
Embeddings only represent, like, one universal one, if that makes sense. So credit cards are actually clustered in this, like, really small area, and this means search works bad. So, like, to give you a concrete example, if you take these two documents, one's from Visa, one's from Mastercard, at least in the system we were designing, like, if you
- 15:44
search something that's about a Visa query, you should never receive Mastercard. But they're all so close to each other that they're actually, like, completely all jumbled together. And this is just, like, a problem with all conventional embedding mechanisms.
- 15:56
So we built this new model that lets you feed in some, like, surrounding documents. So, like, to give you an example, this is kind of the first half of our model.
- 16:04
We would feed in a bunch of credit cards. I guess I put Amex, but there actually was no Amex when we did it. And, um, and the model kind of works like this.
- 16:14
Like, when it produces the embedding for the text, which is here, it also looks at a bunch of surrounding documents. So it can kind of know, like, okay, this text is about Visa, but also all the other documents are about either Visa or Mastercard.
- 16:26
And it gets trained so that it can, like, dynamically adjust the embeddings based on, like, the surrounding context. So I thought this was cool, and it works better. So, like, in this Visa, Mastercard case, the similarity between a Visa and Mastercard is now point one four four, and I think anything containing Visa has a much higher similarity.
- 16:46
So that's, like, maybe correcting one small thing. Um, it works better on, like, out-of-domain stuff. So we have a... I forgot what the climate data set is, a data set of arguments, a data set of financial questions, and then I think, like, scientific articles.
- 17:02
And I guess the point I'm making here is that if you do this contextual thing, embeddings work a bit better. So, like, if you build them in a way that they can dynamically adapt to the domain, they can solve some problems.
- 17:13
But I think at the end of the day, they're still embeddings. And so you're-
- 17:17
Before you move on.
- 17:18
Yeah, yeah.
- 17:19
Uh, was this approach picked up by anyone else that you know of?
- 17:23
Yeah. I think we know they're using it at OpenAI, Anthropic, like behind the scenes now that embedding models are contextual. It's a pretty... It's kind of a free lunch.
- 17:32
Like, you add these extra tokens. Uh, I guess it's, it's kind of hard to build. Like, you have to build this two-stage model, and then, uh, when you embed something, you have to grab some embeddings from the surrounding documents.
- 17:45
But once you build it, it just works, you know, better on like, especially on long-tail stuff. I think if you look at, um, like MS MARCO, which is this large web scale embedding task, it, it really doesn't get much better when you add surrounding stuff because, like, it's already pretty global, if that makes sense.
- 18:03
But if you look at, like, really niche things, the embeddings work a lot better. So yeah, I, I know it's productionized at some other companies. Um, I think if you're actually building an embedding model at your company- And you wanna put effort into making it better, this is probably, like, the easiest way besides data.
- 18:19
Probably the first way is data. Um, there's some recent work that I think is worth mentioning about, like, fundamental limitations of embeddings and vector databases and RAG, which says that, like, if you...
- 18:32
It's not even really worth explaining. But there's, like, some, uh, there, there's some relationships that cannot be captured in a fixed dimensional vector. Like, you have to reason about things to answer all possible tasks, and this is this kind of combinatorial setup where there are so many possible relationships that the embeddings simply can't store them.
- 18:51
And so, like, in theory, embeddings are obviously not the best way to do all possible relationships between text. But I think everyone knows that RAG has issues. Like, I'm glad that no one raised their hand when I asked if anyone was gonna, like, really stand up and speak for RAG.
- 19:09
And, like, we can... I, I, actually think this is a hard point to make. Like, everyone kinda knows this, but it's hard to come up with examples that retrieval can't solve in practice.
- 19:18
Like, speaking as someone who's recently sat down and tried to make benchmarks for tasks that I care about, it's hard to express questions that require kind of this, like, latent reasoning over multiple documents in a way that RAG doesn't solve.
- 19:34
But they do appear. Like, um, anything that kind of requires association between multiple things, or questions that are, they're, like, sort of implied but not explicitly answered by the documents, are just not solvable by current techniques.
- 19:48
And also, if you have interesting examples of this, would love to hear after it's, after the presentation. Um,
- 19:55
hopefully I've made my case that I think RAG... Oh, yeah, yeah, go ahead.
- 20:00
W- I'm curious if you would classify agentic search as RAG as well?
- 20:04
Yeah, that's a good question. So I guess-
- 20:06
Or do you have to-
- 20:07
Agentic search is, like, a model that can grab, and it makes a bunch of queries in a row, and then it responds. Um,
- 20:16
yeah, that's, that's a really good question. I think,
- 20:20
I think I wouldn't classify it as RAG, but I think it has different fundamental limitations that are also tough to overcome. Like, what you, what you would really want is, like, a model that reads the entire thing and reasons about every possible relationship, and then answers.
- 20:36
And I think, in theory, maybe you could build an agentic RAG system that does that, but it would be very expensive.
- 20:43
Because isn't that, isn't that a, isn't, isn't deep research in the direction of that, where it, like, goes through and it pulls, like, hundreds of thousands of sources, but then what ends up in context is only, like, a small subset of those?
- 20:55
Yeah, yeah. I actually think deep research is, like, really in the right direction. Like, they're trying to do something that's a little bit higher level and requires a lot of compute.
- 21:05
Like, I think, um, anything that works better than RAG is going to be more expensive. And so, like, just the property that it takes a while, and it makes a lot of searches, and it thinks a lot is, like, good.
- 21:17
I think that there's probably a more elegant way to train, like, a really big kind of deep research-esque system, but I think that's, that's actually, like, a good way of doing this and, and not the one that I'm talking about today, but it's very promising as well.
- 21:33
Like, maybe the question is like, are you willing to spend a lot of money at training time or at inference time? And deep research is, like, kind of they don't spend a lot of money to train it, but it's willing to wait for a long time at inference.
- 21:44
And I think the things I'm gonna talk about today are more like, if you're willing to spend a lot of money up front, and you get a really smart model that knows all your data already, um, and it's really cheap to do inference.
- 21:55
So it's, like, kinda different sides of the same trade-off. And I think, like, a good way of thinking about these things is, like, to get better models, you're gonna need to pay somewhere.
- 22:04
You know? Like, you're either gonna need to, like, generate better data and spend more time on the data, or you're gonna need to spend time on training, or you're gonna need to spend time on inference.
- 22:11
And the nice thing about RAG is it kind of just works, but anything better will cost more. Yeah.
- 22:17
Uh, getting back to your example of Mastercard versus Visa-
- 22:20
Yeah, sure.
- 22:20
I, I, I don't know if, if that's in your presentation later, but what are your thoughts on using knowledge graph for that as kind of augmenting?
- 22:29
That's a good question. Maybe ask me after. [laughs] I have to think about knowledge graphs. It's been a while. Um, so let's talk about how to learn things in weights.
- 22:37
Um, I think, like, the question that we wanna get at is, like, okay, so say we have the example I showed earlier. Or, like, you have a small data set you've collected from your own personal work, and you wanna teach it to the model.
- 22:49
It's one thing to put it into context, and that's a good way to get started, and if you don't have that much data, that'll get you pretty far. But I think we can do more.
- 22:58
Like, there's some questions that even when your data is in context, the model can't answer. And so what I want us to think about is, like, how can we inject things into a model, uh, in such that it learns better than in context, and also that it doesn't forget everything that it already knows?
- 23:13
Um, I wanna point out something from my own research, which is that there is a fixed capacity to language models. Like, one way to think about this is ChatGPT has, like, only so many parameters.
- 23:23
We have this measurement that it can store three point six bits per parameter. So like, uh, I think a billion parameter model is like, at three point six bits is maybe, like, four terabytes.
- 23:38
Is that right? Four gigabytes?
- 23:40
Yeah.
- 23:40
What it... Yeah. Thank you, thank you. Um, this is, like, some information, but it's actually not that much. So the models, they basically do their best to fit the training distribution, and they throw everything else out.
- 23:53
So, like, to give you a concrete example, this morning I was putting this together. I asked Claude, "What is the capital of the smallest province in Tajikistan?" And it gave me a very detailed answer.
- 24:03
It's actually very impressive. No web search, the model just knows this in its parameters. I guess I'm arguing that this is bad. [laughs] Like, if you wanna build a system that can answer really detailed documentation questions for your company- You don't need it to know what the capital of the smallest province in Tajikistan is.
- 24:21
And since we know these models have fixed capacity, I think that this is bad. Like, what we really want is to know how to like find this kind of thing and just like delete it and replace it with the things we care about.
- 24:32
And I think that's like what we're getting towards, but we don't a hundred percent know how to do that again. I'm sorry. So when I originally put this talk together, the way I was thinking of explaining it is calling it a neural file system, and then I decided to just call it weights.
- 24:44
I think it's easier to understand, but this slide still says neural file systems. Um, so I think there's a few questions here. Like, we wanna train all our data into the model.
- 24:54
One question is like, how do we train it? Do we do RL? Do we do SFT? Uh, what's, what even is the data? Um, another question is like, out of, uh, all the possible data, what do we use?
- 25:06
Do we just like fine-tune directly on our data? Do we try to generate more? I think my argument is that we should try to generate more, and I'll show you why.
- 25:14
And then there's an architectural question. Like, I think for a long time people really cared in the machine learning, deep learning community about like what architectures we should use.
- 25:24
And then for like, what, eight years, everyone who knows what they're doing has really just been using transformers unless they're trying to make them better. And I think now in this world where we're trying to train stuff into models, like, like if you think of, okay, a world we all, each of us have, has our own model
- 25:40
or maybe multiple models, and those models are getting updated a lot, I think we start to care about architecture again. And I'll, and I'll tell you why and like what I think the options are.
- 25:50
So first let's talk about learning. Um, so I think like m- the mental model here, which I mentioned before, is like we're trying to train the model to learn the data as best as it possibly can, and it's gonna be expensive.
- 26:07
So like we didn't like RAG, but also RAG didn't cost us very much money. I think to do better than RAG, we're gonna have to like pay some GPU points, and that's just like the state of the world.
- 26:19
Okay, fine. So this is our model. It's like this homogenous blob of data, and this is our data. So like maybe we have the Mastercard data set, or maybe we collected data about ourselves, or maybe I, uh, collected all my traces from coding in November and December, and I wanna like train the, the model to learn my
- 26:38
problems better. What do I do? How do I actually do this? Um,
- 26:43
let's, let's like start with the dumbest possible approach and just like see what happens. So say, uh, we start with a data set,
- 26:51
and we just train on it. Um, like using I guess next token prediction. So we actually ran this little experiment. This is like, uh, 3M, it's a company that made duct tape. [laughing]
- 27:06
And um, this is like some financial reports. So maybe like you're working there, and you really don't wanna read all of this, so you just wanna ask the model to like really understand this and be able to answer questions.
- 27:19
And like RAG isn't really working 'cause it's like this weird structure, and there's a lot of ways the documents interrelate. Okay, cool, so we're just gonna like train the model using next token prediction, see what happens.
- 27:31
You know what? Actually, even if you don't train the whole model, um, you, you still get zero loss. So the model can perfectly memorize this entire, uh, 3M 10-K financial report.
- 27:43
Um, it's extremely impressive. Okay, so now let's talk to it. So, so we did this, and then we didn't wanna ask anything that's like exactly present in the document 'cause we wanna see if the model's actually good.
- 27:54
So we started, you know, like everyone loves to test poems, so we started with a poem. We said, "Can you write a poem about 3M in fiscal year twenty twenty-five?"
- 28:04
So register your bets, and what do you think happened?
- 28:09
It's terrible. It's terrible. Someone said it. It says, "The passage of a passage is a poem." [laughs] End of sentence. It's crazy. [laughs] Yeah. So now maybe we ask like, why does this happen, and how do we fix it?
- 28:23
So unfortunately this doesn't work, and I actually think this is like one of the reasons why people haven't been doing this yet, is because the dumbest possible approach usually does work in machine learning.
- 28:32
But in this case we have to do something a little bit more sophisticated. Um,
- 28:37
so maybe take a second and think about like what you would do maybe if you're facing this problem at work or in a side project. Um, I think there's like two things we need to fix.
- 28:46
One is that, um, the data is not, it's not exactly what we want to train on, I think. And two is that we probably don't wanna update the entire model because what we did there was basically overwrite all the, you know, stuff about Tajikistan and everything else that's in the model with just like this 3M knowledge, and
- 29:07
I think that's like too specific, and then the model is just obsessed with 3M, and it'll only produce exact copy sentences from the document. That's, that's clearly too much.
- 29:17
So I think we need a better way to update the model, and we need a better way to change the data.
- 29:24
Um, there's this pretty relevant work. I don't know if you follow this like LLM chat thing from Andrej Karpathy. Shout out. I think it's very educational. And he had a really good question, which is like he built this small LLM and trained it from scratch and everything, and then he wanted to teach it about himself.
- 29:40
And okay, maybe the first thing you would try is RAG. You put like a little database of information about yourself, but that's only scalable to a certain amount, and then the model can't really like combine things.
- 29:52
It can only kind of regurgitate facts. And so he wants to actually tr- teach it properly, he says, meaning in weights. And so notice he doesn't just like take one example and, and train the model using next token prediction.
- 30:06
He does something a bit more complicated. He like generates this task. Or you, you don't have to care about the specifics, but there's like basically he makes a diverse training data set of examples that look like the thing he cares about and then trains on it.
- 30:19
And if you go, you can find this. It actually does work pretty well, which is cool. So he's able to teach a novel behavior to a model by like generating a lot of synthetic data that looks like the example he cares about and then fine-tuning the model for a little bit, and it, and it learns There's a
- 30:34
paper that's really good, uh, that's from last year from some folks at Stanford called Synthetic Continued Pre-Training, and they have the same problem. So they have like a really small data set, and they want to teach the model to the data set without like bricking the model, essentially.
- 30:49
And they have this kind of fancy way of generating synthetic data by extracting entities. But I think the important part is that they take a small data set and they generate like a very large, more diverse data set representative of the thing that they care about.
- 31:05
And this is something that like breaks the whole like conventional machine learning paradigm. Like, they only have a small training data set, so, uh, what you learn in school would tell you that you would just like overfit and there's nothing you can do.
- 31:18
You just have to go back and collect more data. But actually, because LLMs are so good now, we can do this second thing where we generate like a much larger training data set that really contains only the like facts that were present in the original data, but it's so large that you can train a model on it.
- 31:34
It's like very strange and only recently started working, but it does work. I'll show you some evidence. Um, the green line is what happens when you do the dumb thing that we tried before.
- 31:44
So you just like fine-tune the model on the data. It actually starts at the black line, so surprisingly it actually gets worse. So it like memorizes the data so well that it can't answer any slightly different questions about it.
- 31:55
Um, the thing they do, they have like two different ways of doing it, but it's basically like generating lots of synthetic data that describes the things in the original data set.
- 32:04
It works very well. Like at, at some scale, I guess a hundred million tokens, close to a billion, they can actually outperform GPT-4 on this data set, which is really cool.
- 32:14
So I think like the takeaway here is
- 32:17
even though you don't have a lot of data, if you're willing to generate like a large synthetic data set that describes the data you have, you can actually train a model on it, and it works really well.
- 32:28
There's a bunch of other papers that do this. One is called active reading. Um, they basically ask the LLM how, what types of things should we generate, and then they generate from it.
- 32:37
There is self-study, which is from this cartridges paper, which is more like question answering, like asking the model to like quiz itself. And then there's this rephrasing the web thing.
- 32:47
I didn't realize my... Whatever. A re-rephrasing the web thing where they kind of like rephrase this entire pre-training data set. So this actually works at scale in kind of a surprising way.
- 32:58
Um, and there's a lot more work in this direction, so I'm really excited about this. Like, and I'm kind of monitoring it. There's a company called Datology that's doing this really well.
- 33:06
They're like generating really high quality synthetic data. It's just like not something that used to be possible until very recently when LLMs crossed some threshold that they're like able to generate data that's good enough to actually train themselves on.
- 33:20
Oh, there's actually something pretty cool that's not in the slides called self-adapting language models. S-self-edit. It's called SEAL, S-E-A-L, and they, uh, ask the model what data to generate to make itself better.
- 33:34
And under some like constrained scenarios, this is actually working. So that's like actually quite bizarre. Um, and like obviously doesn't work infinitely or else they would've caused an intelligence explosion. [laughs]
- 33:45
But the fact that it works at all is like really remarkable and I think like worth monitoring. So
- 33:52
in conclusion for this section, we want to train things into weights. We can generate large synthetic data sets that describe very pretty small data sets, and it works fine.
- 34:02
Um, now I think the money question here is like how do we inject the information into the model? I think before I mentioned we were training all the parameters, and we tried it and it worked really bad, and this is a, a problem that's been around for a long time, and it's called like catastrophic forgetting.
- 34:19
Um, even in old school machine learning, like you train a model to recognize handwritten digits, and then you train a model to recognize house numbers, and it's no longer able to recognize handwritten digits.
- 34:28
This is like a very well-known problem. There's a lot of like theory and like approaches proposed to solve it, but no one really knows how to solve it. It's very, very hard.
- 34:37
Um, but I think there are some easy ways we can get around it in the conventional paradigm where we have like this big pre-trained ChatGPT transformer. Uh, instead of retraining the entire model, there's a few different ways we can do it.
- 34:51
I mean, the first one is retraining the entire model. So the things we're training, I'm highlighting in blue here. Uh, it's like if we take our transformer and we update all the parameters, we're probably gonna forget stuff.
- 35:02
Um, there's another one that's pretty cool called prefix tuning, where you just train the KV cache. Um, I mean, I'll like skip the details for now, but ask me if you have questions.
- 35:12
Prefix tuning is cool. Um, another way is since a lot of these models are called like mixture of experts and they have this MLP layer in them, you can add another part to the MLP that is optionally routed to and used, and that's like pretty scalable.
- 35:27
I think people try this. Um, there's another approach where, where you replace instead of like another MLP, you build this thing called a memory layer, which is like a big lookup table.
- 35:37
I think memory layers are really good. And let me pause and say now this part of the talk is getting close to purely speculative. [laughs] This is like the things that are like they exist and like someone's gonna do this and someone's gonna use like one of them, but I really don't know what the right answer is.
- 35:52
Um, another one is called LoRA, so Low-Rank Adaptation. You probably heard of this very like hot topic. Um, they kind of like train a small, a small matrix or small few matrices to adapt the linear layers.
- 36:06
So it's like if your model's ten billion parameters, maybe you train ten million parameters that can like control it. Um,
- 36:14
and if we look at them together, maybe it's not super obvious which thing would work best. Like ICL is just like putting stuff in context. So we have in context RAG, full fine-tuning.
- 36:25
We could do the memory layers in, in MLP. Prefixes, which is prefix tuning. We could do LoRA. We could also do add something to the mixture of experts. I think to me it's not like clear, and I'm not positive that it matters which one we do.
- 36:38
Like, I think the main thing is like we have this giant model and we're adding a tiny bit to it to control it and train only those parameters. That way we retain most of the information in the model.
- 36:49
I think that's like the most important part. But I think for the end of this talk, I'll just talk through, like, what I think people are doing in this space up to, like, the minute, and then you can make up your own mind what you think the right way to do it is.
- 37:03
So let's talk for a second about what properties we want. I think we want, um, we want our changes to the model to be very small. Like, say you're serving a model to each person, you actually can do it, but you have to use one of these, like, parameter efficient methods.
- 37:17
If you're trying to fine-tune a new Kimi for each person, Kimi's like a terabyte. It's a trillion parameters. It's just, like, not even storable, let alone servable. Um, we want something that's resistant to forgetting, like we said.
- 37:29
So it would be nice to have an architectural change that's both small and makes the minimal impact on the model as it is now, because the model as it is now works really well.
- 37:39
Um, and preferably high capacity. I think, like, changes that are really expressive and can capture a lot of facts in few parameters are the ones that we prefer, and we wanna be able to do inference quickly.
- 37:51
As, like, a small aside, you actually can do this quickly with a lot of, um, a lot of these methods. Like maybe some of you have seen Tinker, this new training API from Thinking Machines.
- 38:02
It's basically all predicated on this idea that you can, you can serve one model per person as long as you do LoRA and batch the LoRAs. And there's like...
- 38:12
It's actually most interesting from a systems perspective. There's, like, ways you can train it and train each one separately, and there's ways you can do inference, and it basically has no cost, um, which is really interesting just 'cause, like, the base model doesn't change, and we all share the same base model.
- 38:25
So all the ideas I'm gonna b- talk about are kind of, like, in the same direction as Tinker. Um,
- 38:32
we can think about, like, whether certain methods might learn more or forget more. Um, so this is comparing LoRA to full fine-tuning. So LoRA makes a tiny change to the model.
- 38:44
Full fine-tuning updates the entire model. And on two different settings, they show, like, LoRA here is, like, purplish or pink. The pink one's a little bit smaller capacity. Um, it basically doesn't do as well, at least when you're doing SFT.
- 38:58
Uh, LoRA can learn a little bit less, but also if we look at how much it's degrading, it forgets less. So this paper's called Learn-- LoRA Learns Less and Forgets Less, and it's, it's actually a very nice finding.
- 39:11
So, like, if you wanna at least teach a model via SFT, and you use one of these low-rank or parameter efficient methods like all the ones I described, they're gonna make a small change to the model in a way that it's probably not gonna be as expressive as full fine-tuning, but it also doesn't destroy a lot of
- 39:26
the knowledge. Um, here's something going the exact opposite direction. This is the result from Thinking Machines showing that they think LoRA is about as good as full fine-tuning, which is interesting because they're doing RL.
- 39:40
So it's, like, maybe dependent on the training mechanism. Like, if you do RL, maybe it makes small updates, and, um, you can do LoRA, you can do memory layers.
- 39:50
But for SFT, it, it really has to store a lot of information, so you really have to do full fine-tuning. I think that's the takeaway I have. And I have some...
- 39:57
actually a paper that's, like, kind of blocked for legal reasons, but coming out soon. Um, here's one result from my paper that's relevant to this. So we have this, like, tiny LoRA thing that's even smaller than LoRA.
- 40:09
And, well, there's actually LoraXS, which already exists, and then we made tiny LoRA, which is even smaller. And if you're doing RL on GSM8K math reasoning, you can train fourteen parameters and get, like, ninety-one percent accuracy, which is pretty crazy.
- 40:25
I think, um, there's, like, a lot of reasons for this. Like, RL makes really tiny changes. I think this Qwen model, like, is something fishy is going on with the training data.
- 40:35
Do you have a one parameter experiment?
- 40:38
Oh, yeah, yeah. This is one parameter. It actually learns. It gets five percent better with one parameter. [laughing]
- 40:45
Pretty cool.
- 40:45
That's amazing.
- 40:46
Yeah, yeah. It's, it's, it's really nice. I think, um-
- 40:50
Literally the smallest.
- 40:52
Yeah. [laughs] Yeah, the smallest thing you could possibly train. It's more like you, you generate a lot of random projections, and then you control them all with one number, if that makes sense.
- 41:01
Like, the model actually changes a lot, but the only thing you can actually train and store is the one parameter.
- 41:08
Uh, I'll tell you more about it later. Um, but yeah, it's pretty cool. Um,
- 41:14
this is another result that's, like, kind of in the mix, but I'm not sure how to place it. So if you do the KV cache tuning or prefix tuning, this paper thinks prefix tuning works much better than LoRA.
- 41:25
I met some people in Meta, um, when I used to be affiliated there, that said that they think LoRA works much better than prefix tuning. So I really don't know.
- 41:32
But I think, like, what it really will come down to is, like, when you do it at scale, what's, like, most efficient. And I'm not exactly sure, but I think prefix tuning is a pretty good candidate because, like, KV caches are so commonly used these days, and, like, a lot of the system stuff is built around KV
- 41:50
caches. I think a cool thing about Thinking Machines is, like, they're designing this entire organization around, like, scaling LoRA, which is awesome, but it's not really possible in open source right now.
- 41:59
Like, there's not kernels for training many LoRAs at the same time. It's, like, very complex, and you have to have a lot of people working on that. Prefix tuning, on the other hand, is, like, very well supported.
- 42:08
Um, and then finally, I'll quickly talk about memory layers. This is another approach to injecting data into models, which I think is good. This is like, uh, adding a expert to the MLP, but the expert is just, like, this giant differentiable lookup table.
- 42:23
So it's kind of not that important exactly how it works, but it's like, it's just a different way to inject information into models. The cool thing about memory layers is it's controllable.
- 42:33
So in this work, uh, by Justin Lan from this year, they specify exactly which parts of the memory layer get updated and keep it to, like, a very small number.
- 42:45
And so their result shows that memory layers actually work the best. So memory, the, the, the axes here are forgetting, so down is bad, and learning, right is good.
- 42:56
So the memory layers basically don't forget at all. And they learn close to as much. So I think if you're trying to inject information into models, if you really care about them not forgetting any of their base information, maybe memory layers are the way to go.
- 43:11
I think honestly, there's a lot of conflicting evidence right now. Like, some people think LoRA is good, some people think prefix tuning is good. These people think memory layers is good.
- 43:19
I really am not sure, but I think it's gonna be one of them.
- 43:23
Okay, cool. That's, that's the end of the training stuff into weights part. Maybe... Actually, I'll stop and see if anyone has any questions about the different parameterizations. Yeah.
- 43:32
Can you go back to the slide where you were showing the, uh, when, uh, GRPO-
- 43:40
Oh, yeah, yeah, yeah. From, from my yet unreleased research.
- 43:44
So have you used SFT before?
- 43:47
Yeah, yeah. I can show you the SFT results later, but SFT, uh,
- 43:53
takes a lot more parameters in the short explanation. Like many, many more, like a thousand X moment or something.
- 43:59
A- and you a- attribute that to the sparsity of the reward?
- 44:02
Yeah, yeah. I think it's something like that. Like the SFT learning signal is like cross entropy on all of the tokens with or without thinking tokens, and that's a lot of bits essentially.
- 44:13
And then RL just gives you a one or a zero. If you get it right, and you already knew, then it's, uh, no information. If you get it wrong, you get like one bit.
- 44:21
So I think because RL is like so sparse and, uh, information efficient, then you can do it with way fewer parameters. That's, that's kind of the takeaway from our paper actually.
- 44:30
So you didn't do GRPO after doing SFT?
- 44:34
No, no SFT. We just either do GRPO or SFT, and then we see like kind of how many parameters you need to train to get to equivalent performance, and SFT requires many more parameters.
- 44:48
Uh, so here we are comparing the like, uh, uh, training versus RAG, like we are being... We want to solve the problem, what we are facing in the RAG.
- 44:57
So is the volume of the document also matter? Like if you have any studies like, uh, uh, because if, if some problem has a less number of document, uh, RAG will be better or the, uh, training will be better?
- 45:11
That's a really good point. Um, maybe that... Let's, uh, go to the last slide. So I think the question is like, okay, if you're trying to train all of your data into a model, but something only happens once.
- 45:23
Yeah. Means when, when I should pick, uh, focus on RAG and when I should focus on like, uh, uh, like a training, like...
- 45:30
Yeah.
- 45:30
Because every time means I have like a small, uh, set of the document, the training might not be feasible.
- 45:37
Yes, yes. Like it... You're like maybe you... Something is so underrepresented in your data that it probably wouldn't be-
- 45:45
If my data is frequently changing might be-
- 45:47
Your data is changing a lot. Yeah. Maybe in the short term, it's hard to train. Um, yeah. So let me point out like, okay, so obviously we're always gonna put stuff into context, and I think we'll also probably always do RAG.
- 46:02
Like I think, um, there's basically no scenario, uh, that you can imagine for a long time where you're just like always training the model and never doing RAG. I think you'll do both.
- 46:12
I think like maybe if you have a ton of documents, I don't know, maybe every day you do this big training, and then every time you serve, you also do RAG.
- 46:19
Yeah.
- 46:20
And so like what I really imagine is like... Or maybe my, my point is that no one is doing this right now.
- 46:26
Yeah.
- 46:26
And like people will start doing it.
- 46:28
Do you have, do you have any like a projection like after a certain amount of data, like, uh, training will be like a more efficient than the RAG, like when-
- 46:36
Yeah. Yeah, yeah. No, that's a really good question. Uh, no. Like I think, I think this kind of thing is really new, so there's a lot of room for analysis like that.
- 46:43
I would definitely be interested to see both analysis on how the frequency of information affects like the trade-off, and how just like how much data you have to have for training to become economically feasible.
- 46:55
That's a really good question. Yeah.
- 46:57
Um, is your suggestion kind of in, um, diving more into like the weights side of, uh, the presentation to use a fine-tuned model for like completion type tasks, or also for embeddings?
- 47:14
Oh, yeah. That's a good question. Um, no, I think, I think, uh, the fine-tuning I'm talking about is all for like assistant agent completion.
- 47:23
Okay.
- 47:23
Um, it's an interesting question. You probably could do like dynamic embedding model training, but I guess like the way I think about it is like the real like 10X improvement here is gonna come from training into weights.
- 47:35
You could maybe make RAG like two X better if you really, really work, but I think there's so many fundamental problems with it that I wouldn't spend that much time on making embeddings better.
- 47:46
What were... What do you feel like the most fundamental problem is, where even if like your retrieval was fantastic-
- 47:52
Mm-hmm
- 47:53
... you'll still kind of-
- 47:53
I think like chunking, like, um-
- 47:55
Okay.
- 47:56
You just like kind of retrieve some of the stuff you need, and then you can't really reason across all of it.
- 48:01
Makes sense.
- 48:01
And like I think in the limit, like there's some types of data where like no matter how you chunk, you'll never get like everything you need.
- 48:08
Mm-hmm.
- 48:08
If that makes sense.
- 48:09
Yeah, totally.
- 48:10
Cool. Yeah.
- 48:12
Do you see any fundamental limitations as you scale up the amount of personalization you need? Let's say you had a B2C product that had a hundred million or ten million users, you want a memory for all of those.
- 48:22
Mm-hmm.
- 48:22
Do you think that's just not feasible, uh-
- 48:24
You say ten million users?
- 48:25
Yeah, ten million or a hundred billion, somewhere in that range.
- 48:27
Yeah. Um, no, no, I actually think it is, it is feasible. Like LoRA, maybe you train a, a few megabytes per user or something. It's not that crazy, right?
- 48:38
Like YouTube probably has gigabytes per user.
- 48:41
But your update model is hard.
- 48:43
Right. That's a good point. Like the continual updates are hard. Like probably in realistic short term, it's more like you update once a day or something like that. But I think that's, that's doable.
- 48:52
But you made a good point that the paradigm I'm describing is much more expensive than current-
- 48:57
Also, did you consider there's a lot more that you can do using the other two left buckets. You can press the data with context, you can press it before you put RAG, right?
- 49:04
That-
- 49:05
Buckets, you don't just have to use RAGs and use SQL and knowledge graph, all of them together in different buckets, and that solves a lot of problems.
- 49:13
Yeah. Yeah, that's a good point. There's kind of like three axes of optimization here, and I guess, like, we are-- we're getting pretty good at this. We're okay at this, and we're horrible at this, and so, like, we'll continue improving upon all three axes.
- 49:28
Yeah.
- 49:28
What's your, uh, like... I'm kind of hearing that maybe it's not defined yet, but what's your kind of like intuition or guess in terms of, like, where the decision boundary is in terms of investing your effort in those optimizations, particularly in like, let's say, a couple of years where you could do something like a deep research, but
- 49:47
it would be way cheaper and way faster. Um, when-- What are their...
- 49:53
Y-you were saying that, that there isn't like a number of documents, but what is the boundary that you would think about looking at? Is it the freshness of the data?
- 50:00
Is it how fast it's changing? Is it number of documents? Or what's your decision process?
- 50:04
Yeah. I, I-- It's a really good question. I th-I think, um, I think the paradigm I'm describing is especially effective when you have like a large amount of data that's not been indexed into the LLM at all, and it gives you a big benefit there.
- 50:17
I think when you start seeing like sparser updates to your dataset where like some new data comes in, but it's not that much, and it's like fairly often, then you probably wanna turn to inference time approaches that are closer to deep research.
- 50:31
Um, yeah, that guy had a question along.
- 50:34
Yeah. Can you elaborate a little bit more about the sy-synthetic data generation? So let's say that you have an LLM, you need to get it to talk, uh, in a similar sort of language and terminology of like a proprietary data, right?
- 50:50
Like, millions of these documents. Like, how would synthetic data generation in that context be helpful?
- 51:00
So your company has millions of documents, you said? And you want the model to-
- 51:04
It's, it's more like a scenario.
- 51:05
Yeah, yeah. Okay.
- 51:06
Yeah.
- 51:06
Yeah, yeah. Um-
- 51:07
'Cause it wouldn't-- 'cause to what you said, you know, you wouldn't just train off of the next word only, right?
- 51:13
Yeah.
- 51:15
You have to, um, try alternatives such as... And I think one of those options that you had talked about was, uh, synthetic data generation.
- 51:21
Yeah. Yeah. No, I think, I think synthetic data generation could work for that problem. So I guess like, um, it depends on how information dense your data is. If you have millions of documents from your company, I would guess many of them share formatting and only contribute maybe like a few bits of kind of global information to
- 51:43
the dataset. And so what you wanna think about is like, does there exist a function that could produce a good training dataset for an LLM that would teach it about my data?
- 51:51
And like, there probably is. Like, you could probably design some strategy that looks at the documents, kind of like figures out what's new about each document, and creates like a question answer pairs.
- 52:01
But this is very blue sky. Like, I think a lot of people are working on this right now, but I don't have like a, a global answer of how to actually do it.
- 52:09
Right now, my only solution that I can think of is, um, you know, getting to generate that Q&A pair, you know, where you just said.
- 52:17
Right.
- 52:17
And then using it for as a training document. I'm just wondering if there's like other ways that you can solve this.
- 52:25
Yeah. Yeah. I think it also depends on what types of questions you'll be asking about the documents. Like what you really wanna model is like all possible questions or something like that.
- 52:32
But I think Q&A gets you pretty far.
- 52:37
Cool. Yeah.
- 52:38
Um, so with, with this approach, right, you, you, you mentioned this example where you're, um, uh, you would train your model, right, on 3M, uh, quarterly earnings, right? Uh, I think ten, 10-K, 10-Q, uh, documents.
- 52:54
What would like... What would the prompt basically look like, right? Like, i-is there, is there anything in-
- 53:01
Mm.
- 53:01
-within like the in context learning that would still need to be s-
- 53:05
Mm
- 53:05
-kind of specified to-
- 53:07
Mm-hmm. Okay
- 53:08
-bring your data into the context?
- 53:12
Yeah. Uh, so I think the question was, if you start with the 3M example we had, and you train all that into a model using something like magic synthetic data, what does actually the prompt look like?
- 53:21
Yeah.
- 53:22
I think actually if you do it right, you don't need a prompt at all. Like, you can just ask the model a question, no system prompt, no g- extra information, and if nothing has changed, it should know everything.
- 53:33
Like, and you even... There's some scenarios where there's only one document, and the model knows which document it is, so you don't have to specify that you're even asking a question about the document.
- 53:40
It's like implied, you know? So, um, it depends on how you set it up, but I think in like the ideal case, there's no prompt at all.
- 53:51
Yeah.
- 53:53
I... It's not obvious to me that information is best stored in model weights.
- 53:58
Yeah.
- 53:58
Why do you have... Do you have that? Um, it feels implied. Do you have-- Am I, am I right?
- 54:04
That's a good question. [laughs] So he said, it's not obvious that information needs to be stored in weights. Yeah, yeah. This is, this is a good question. I think, um, I'm not saying that it's best to store information in weights.
- 54:17
I guess I'm arguing that that gets you a lot, and we're not using it right now.
- 54:22
Yeah.
- 54:23
And like, once you get to the scale of like a GitHub repo, you might have millions of tokens, and it's just like very expensive. And so at least, like this is the cheapest way to do it.
- 54:34
The question of like, can we generate synthetic data to do better than in context is like, it's, it's hard, I think. It's like, that's research.
- 54:44
Do, do you know what I mean when I say it's cheaper though?
- 54:47
Yeah.
- 54:47
Like, if you have a million token prompt, you can just like compress it into the weights and produce a model that gives the same outputs with no prompt, and then the inference costs less.
- 55:00
So how long [inaudible]?
- 55:04
We can talk after. Follow up. Yeah.
- 55:06
I know
- 55:07
Yes.
- 55:07
How many issues that there is no adversarial in data being generated, right?
- 55:13
Hmm, that's actually a really good question. Never thought about it before. Um, I think it's probably pretty hard. Like, I guess if you're training on user data, and like you have some user that wants to sabotage your system, and you're generating training data from their inputs, there probably are a lot of like security risks.
- 55:30
And, uh, I guess in this scenario, if you're serving the same model to that user and it doesn't work anymore, that's like not your problem. But once you start aggregating information across users, I bet it becomes hard.
- 55:40
I'm sure ChatGPT has the same problem, where some people always click thumbs down instead of thumbs up to try to like- [laughing] -stick it to the man. [laughing]
- 55:49
In the research, uh, they segmented geographically across countries because some cultures are inclined-
- 55:56
Hmm. [laughing]
- 55:57
-to, so it created bias in the data.
- 56:00
That's funny. [laughing]
- 56:02
Yeah.
- 56:02
Here.
- 56:03
Yeah. Um, so just thinking maybe a little bit about practical implementations of something like this, um, especially in terms of like say version control, like you mentioned GitHub models that you keep fine-tuning over time.
- 56:14
Say you're a company that just changed a policy, and it's just a one-line sentence, we honor something to we do not honor it anymore.
- 56:21
Mm-hmm.
- 56:21
And that keeps coming back and forth. Do you then, you know, start from the base model again and then fine-tune that?
- 56:27
Yeah, yeah.
- 56:27
Or go back to the one that already a good representation of it, and just has to change that one small thing? And then, you know, how that kind of is joined at the hip with hallucinations, which is kind of why people are doing full RAG now, partially to avoid that.
- 56:40
Do you have any thoughts on how that might work?
- 56:42
Yeah, I think it... So, so his question was about
- 56:46
what do you do once you start making multiple updates to the model, especially when you're up like in collecting information. And I think like the optimal synthetic data strategy would somehow figure this out during training.
- 56:57
And maybe even like if there's some documents from a few days ago that are no longer relevant, you can just like delete them. But I don't know.
- 57:04
It's hard.
- 57:05
How, how we can give more attention in the same, like whatever, uh, let's say, uh, uh, information is conflicting with each other, uh, whatever pre-trained versus what, uh, current document we are giving for training.
- 57:17
If it is a contradictory each other, but I want more preference from my document.
- 57:22
By what we are doing in RAG, like asking the questions from the ground truth. So how, uh, it will replace that, uh, scenario? Yeah.
- 57:33
I'm not sure I understand the question.
- 57:35
Sorry?
- 57:35
I, I don't know if I understood your question.
- 57:37
So, okay. So what you are-
- 57:40
I didn't understand your question.
- 57:41
So my question is like I, uh, we have the da- uh, whatever the training data we are giving each other is, uh, contradicting with the pre-training data. It is a conflicting.
- 57:51
Now, while asking the question, while the inference, I want to give more preference on my data. I don't need the pre-training information. That's why we are using-
- 58:00
Mm-hmm
- 58:00
... RAG. Like I, I need the, uh, output from my ground tr- uh, whatever the context I'm giving.
- 58:06
Mm-hmm.
- 58:06
So how it will, uh, we can achieve in the, like a training?
- 58:12
I think that the, the paradigm I'm proposing has all the same limitations of RAG. Uh, I'm not positive that answers your question, but like, for example, if, uh, like maybe in the scenario he said, where you said something many times and then it turns out not to be true, both RAG would retrieve that, and in the, uh,
- 58:35
dumbest setup, that would also be present a lot in the training data. So I think like the same problems have to be solved. Yeah.
- 58:42
Have you done any work with federated, uh, tuning, fine-tuning, uh, parameter efficient fine-tuning? So what if you were a problem-
- 58:48
Yeah
- 58:48
... with millions of users? Have you done any research in this spot?
- 58:52
No, no, no. Uh, not really, but I think it's an interesting op- opportunity. So like back in the day, a lot of people were really excited about the idea that you could share gradients and train the same model across many machines.
- 59:03
This is federated learning. And I think like one of the problems why it's hard is because the models now are so big that the network costs are way too high.
- 59:11
Yeah.
- 59:11
And because like I'm arguing that you only need to train a million parameters instead of a trillion, it probably comes back into play. So I think it's a very good idea, especially in the RL world, where you do a lot of work for a long time, and then do gradients like very seldomly.
- 59:27
So I think it probably will come back, and it's smart to think of it, but it hasn't quite yet.
- 59:35
Um, maybe I'll take like two more questions. Yeah, go.
- 59:37
Um, so y- your argument here about training in, um, information seems to be, uh, counter to Karpathy's view of like a reasoning engine. Like distilling just the pure, like, you know-
- 59:51
Hmm
- 59:51
... intelligence aspect of a, of a model down to like a two billion parameter thing. Um, uh, a- and like I think that there's a bit of overlap there.
- 1:00:00
Like-
- 1:00:01
Yeah
- 1:00:01
... um, uh, like a lawyer is not, doesn't have the entire legal code memorized, but they know how to use the tools available to them to find what they need to.
- 1:00:12
And so I, I think part of it is kind of a combination of those two things, where you're doing task-specific training with something like this on a relatively small reasoning brain to get a sense of where it needs to find the things that-
- 1:00:28
Mm-hmm
- 1:00:28
... uh, might become stale or, or, you know. Am I on the right track here?
- 1:00:34
Yeah. Yeah. So I think there may be comparisons between some people who have said, "Oh, the best model we could ever have is like really small and knows nothing, but can use tools really well," or something like that.
- 1:00:45
And I guess I, I was proposing some similar ideas. I said, "Models know way too much." I think everyone agrees the model doesn't need to know the capital of the smallest province in Tajikistan for most use cases, at least in like my w- life.
- 1:00:59
It doesn't need to remember, you know, encryption keys or-
- 1:01:01
Yeah. But I think there's... I, I think that's a very philosophical question, but, um, I think it's really hard to create a model that doesn't know anything. And so I'm more advocating for like specialized models that are good at something you care about, but bad at other things, rather than advocating for a model that's like bad at
- 1:01:19
everything or something. Uh, okay, last question. Yeah, you.
- 1:01:22
Yeah. Have you done any research yet into the temporal elements of the information?
- 1:01:26
No, but I think that's like one of the first things to think about is like, okay, if you have information from day one and day two and day three, do you just sort of like concatenate everything, or do you train in order, kind of like you were asking, or do you like train multiple models and merge them,
- 1:01:40
or... I, I actually don't know. But that's a good segue. So now I'm, uh, I'm working on this, uh, problems related to this a lot, thinking about this a lot.
- 1:01:50
Um, started a company with a few other people and, um, this is like the kind of research we're doing. If anyone knows someone who lives in San Francisco and is a good engineer, and you think they're interested in this, let me know or send me an email.
- 1:02:04
Or, or if you're interested in like using this kind of thing, you send me an email. That would be great.
- 1:02:08
It, it's temporal stuff or-
- 1:02:10
Not necessarily. I mean, it, it's kind of all of this, I would say. Um, trying to build models that you can teach things to.
- 1:02:17
Tell us more. [laughing]
- 1:02:19
No. All right. Thanks so much for having me. This was great. [clapping] [upbeat music]