AI Engineer World's Fair 2024
The Hierarchy of Needs for Training Dataset Development
Read the talk
Training Data Needs an Evaluation Loop—and Storage That Can Keep Up
Dataset development connects evaluation, selection, and human judgment to an infrastructure problem: filtering, shuffling, and streaming large samples without slowing research.
From a talk by Chang She and Noah Shpak
Before you start: Familiarity with model training, SQL, and basic storage concepts such as rows, columns, and object storage will help.
What should the model train on?
What should a model train on, and how can a team tell whether its choices helped? Answering those questions requires both data research and data infrastructure. Chang She introduces his infrastructure background through pandas, large data systems, and recommender systems, leading to his work as co-founder and CEO of LanceDB, a database for multimodal AI.
At Character.ai, the question connects directly to a consumer product. Noah Shpak describes leading its AI data platform, supporting both internally trained foundation models and the service people use. Training choices must produce results on academic benchmarks, but also in A/B tests and user engagement. His team builds tools to accelerate that research. A suitable storage format matters because researchers need to inspect, change, and reuse the data they train on.
The level of inspection changes across training stages. Pre-training tends to begin with broad composition and quantity; post-training demands a closer look at individual tasks.
| Decision | Pre-training emphasis | Post-training emphasis |
|---|---|---|
| Composition | Domains such as books and chat | Specific tasks and their contexts |
| Scale | Model size and token requirements | Coverage of relevant examples |
| Granularity | Broad domain mixture | Difficulty of a math or multiple-choice problem |
These emphases overlap. The practical difference is how much detail researchers need to understand about each example, across the whole dataset.
Several research problems cut across that distinction: achieving good results with less data for a similarly sized model, choosing useful metrics for sampling, and measuring diversity. Diversity is especially difficult: having many examples does not by itself establish that they cover meaningfully different behaviors. These questions motivate a hierarchy of tools for understanding and improving the training set.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build from clean data and evaluations
Clean data and evaluations form the base of dataset development. Shpak’s team moves quickly from cleaning to evaluation because it needs a compass for deciding whether a change helped. As post-training work expands, managing datasets becomes a research requirement in its own right. A mixture is a collection of datasets together with choices about how they enter training sets and batches. Knowing that one component is Wikipedia is not enough; researchers need to know what examples it actually contributes.
That need leads to analytics. Token counts and sequence lengths describe the basic shape of a mixture. For code, language labels such as Python or Java are only a beginning: task difficulty, the number of functions involved, and the number of classes an answer must generate can reveal more about what the model is being asked to learn. Shpak identifies reading data as a particularly valuable practice. The useful analytics automate distinctions researchers first discover by inspecting examples, model outputs, and performance together.
Above that foundation, language models help improve the data used to train language models. The operations serve different purposes:
- Dataset selection: Match the distribution of available data to desired model behavior. Embeddings, retrieval, and clustering help find examples relevant to the evaluation goals.
- Quality scoring: Use in-house classifiers to assess desired properties, or use prompted classification as a simpler route than building and evaluating a dedicated classifier.
- Synthetic data: Combine big-data tools such as Spark and Trino with GPU-backed services for prompting, embedding, and classification. This makes enrichment and augmentation part of the data platform rather than a separate manual process.
Synthetic preference pairs, for example, let researchers explore a method before they have its best possible dataset. Their immediate value is signal about which types and shapes of data might work. Shpak explicitly treats synthetic data as imperfect: once the approach looks promising, human labeling can improve it. Human work also improves the classifiers themselves and rewrites flawed synthetic examples or flawed existing data. The hierarchy therefore ends with human judgment refining the processes below it, and the whole sequence motivates tools that make these operations easy for researchers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make dataset materialization part of training
The platform interface Shpak shows is YAML containing a SQL block. That arrangement makes dataset selection explicit at the point where a training job requests its data. Conventional training artifacts—TFRecords or JSON Lines, for example—can become a source of confusion: a job consumes files, produces strange results, and leaves the researcher asking what those files actually contain.
Materialization belongs in the training workflow, with responsibilities separated from the training code. A query expresses the data choice; a materialization service produces the artifacts the job will consume. For a hypothetical table of code examples, a small YAML recipe could express a language-and-length slice like this:
yaml
sql: |
SELECT text, language, token_count
FROM training_examples
WHERE language = 'Python'
AND token_count BETWEEN 128 AND 2048
The particular table and thresholds are illustrative; the important boundary is between selecting and materializing data and running the training computation.
Multimodal data makes that boundary more consequential because both the volumes and the problems grow. The materialization service has a simple external contract: send a request, receive lists of files. Loading those files efficiently remains a separate problem, especially when the dataset becomes large.
Lance provides the random access that lets the loader shuffle references to rows instead of shuffling the rows themselves. For example, a loader can permute row references from [0, 1, 2] to [2, 0, 1] while leaving the stored samples in place, then fetch samples in that order. This does not eliminate reading the samples; it avoids moving their full payloads merely to establish a randomized order. The objective is faster research iteration while keeping GPUs supplied with work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filter, shuffle, and stream require different storage strengths
The loading problem leads into She’s infrastructure argument. AI workloads combine access patterns that traditional data warehousing and analytical systems do not necessarily optimize together. Distributed training provides a concrete example:
- Filter: Scan the raw dataset to select appropriate samples.
- Shuffle: Draw random rows from the filtered set.
- Stream: Move the selected text, images, or videos from object storage into GPUs.
The steps are sequential, but each puts a different demand on storage.
| Training step | Required capability |
|---|---|
| Filter | Fast scans |
| Shuffle | Fast random access |
| Stream | Efficient handling of large binary blobs |
A format that scans efficiently can still be expensive for scattered reads. A system that retrieves small records quickly may struggle with large video payloads. The training workflow needs all three capabilities together.
She argues that existing formats and infrastructure commonly satisfy only one or two of these requirements well. He calls the tension a “new CAP theorem” for AI data, using it as an engineering analogy rather than a formal impossibility theorem. It motivates the Lance format and, around that format, LanceDB.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Larger rows and faster data creation compound the problem
Multimodal data increases the cost of moving an individual row. She illustrates the progression with rough size estimates:
| Representation | She’s illustrative size comparison |
|---|---|
| Simple scalar tabular row | About 150 bytes per row |
| With embeddings | Roughly 20–25 times larger, depending on dimensions |
| With images | Another roughly 20-fold increase |
| With videos | Much larger again |
These are contextual estimates, not a benchmark with a specified dataset, encoding, or compression setting. Their purpose is to show why moving full rows becomes increasingly expensive as more modalities enter the dataset.
The number of rows can grow faster too. She describes generated data arriving at thousands of tokens per second, rather than being limited by the pace of human interaction. His scale comparison extends to organizations: teams of 10–20 people may manage tens of terabytes through petabytes of data. Together, larger records and faster creation make storage efficiency relevant even to relatively small teams.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine a columnar file with versioned table operations
Lance addresses the storage requirements at several layers. First, it is a columnar file format: like Parquet, it supports scans, while also targeting efficient lookups of selected rows. The removal of Parquet-style row groups that She describes belongs to the Lance v2 design, which separates column pages to accommodate wide values and inline blobs. This was a design in transition at the time of the talk: the later Lance File 2.1 retrospective dates v2 becoming the default to fall 2024.
Second, Lance provides a lightweight table format. Added data is automatically versioned, and new columns can be added without copying the original dataset. That is particularly useful when the original columns contain large images or videos: an experimental feature column should not require rewriting those payloads. She calls this zero-copy schema evolution and connects it to trying features and rolling them back later.
Time travel provides another recovery mechanism. If a change introduces bad data or an error, the table can return to a previously known good version before the problem propagates into downstream training. She describes that rollback as instantaneous; the talk does not specify conditions or a latency measurement. The practical benefit is that researchers can revise dataset state without treating every experiment as an irreversible rewrite.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An index must make the selected rows cheap to fetch
The third layer is indexing. An index can identify which rows match a request, but that only solves half the problem: storage must still retrieve those rows efficiently. She contrasts Lance with Parquet on this basis. The distinction is performance, not the absolute presence of random access: Parquet’s optional page indexes support row-index navigation and selective page reads, while implementation support and page-level read amplification affect the cost of fetching individual values.
Lance’s indexing extensions cover several access patterns:
- Embedding indexes: She describes billion-scale vector search directly from S3.
- Scalar indexes: Accelerate filtering on metadata columns.
- Full-text indexes: Support keyword or fuzzy search over an S3 dataset.
For these search workloads, She proposes that a separate Elasticsearch cluster can become unnecessary. The broader goal is a single table that supports several kinds of access, rather than a separate data copy for each task.
| Data and operation | Interface described in the talk |
|---|---|
| SQL over metadata or time-series columns | DuckDB, Trino, or Spark |
| Training on blobs and tensors | PyTorch |
| Similarity search over embeddings | Embedded vector index |
These interfaces connect exploration, retrieval, and training to the same underlying table. A researcher can analyze the dataset, search for relevant examples, and use its stored payloads for fine-tuning or training without making each activity a separate storage system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the research loop fast as requirements grow
LanceDB is the database layer built around the format. She claims distributed search across billions of vectors with low latency, high QPS, and an order of magnitude less infrastructure than other vector databases. That is a vendor performance claim without a specified comparison setup in the talk, rather than a measured result readers can apply to their own workload.
The intended scope of multimodal infrastructure is broader than image or video generation. She distinguishes three dimensions:
- Data: Features, audio waveforms, images, and vectors—including text and image embeddings.
- Workloads: OLAP SQL, vector search, full-text search, filtering, and data-frame operations.
- Scenarios: Production RAG, search and retrieval, personalization, model training, and data-lake analysis or exploration.
The same format is intended to connect those dimensions, allowing data to serve operational applications as well as research and training.
Shpak closes by making research speed the strategic priority. His team’s experience is that tools slow down under load and under new multimodal requirements. The platform’s job is to sustain the loop: understand the data, change it, train, evaluate, and try again. Storage choices matter because they determine how readily researchers can keep doing that as the workload grows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and getting-started examples for the Lance data format.
Further reading
Explains the 2024 design for independent column pages, efficient point lookups, and storage without row groups.
A 2024 Python walkthrough for streaming, tokenizing, storing, and retrieving training data by index.
- Apache Parquet page indexesDocumentation
Specification for page skipping, selective reads, and navigation by row index in Parquet.
Updates since the talk
A 2025 paper coauthored by Chang She examining columnar random access, scans, and structural encodings on NVMe storage.
Read the complete timestamped transcript
- 0:00
[upbeat music] We're excited to be here, and we're excited to be talking to you about, uh, training dataset development for LLMs.
- 0:21
So my name is Chang She. I'm the CEO and co-founder of LanceDB. I've been creating data tools for data science and machine learning for almost two decades, uh, starting with being one of the co-authors of pan- the pandas library a long time ago.
- 0:34
Uh, spent a bunch of years in, you know, big d-data systems and recommender systems and most recently I started this company, um, LanceDB, which is the database for multimodal AI.
- 0:46
And these days I spend about roughly equal time, uh, tweeting and, and on GitHub. [laughs]
- 0:53
And yeah, hi everyone. I'm Noah. I currently lead the AI data platform at Character, and Character.ai is one of the leading personalized AI platforms, so we train our own foundation models as well as run a direct consumer online platform.
- 1:05
And I focus on data research, so since we train our own foundation models, we need to learn what we need to train on to engage our users. And so we're focused both on academic benchmarks as well as things like AB tests and trying to get more engagement on our platform.
- 1:23
My team is focused on research acceleration as well, so we tend to build a lot of tools and leading to this collaboration with Lance and how we think about storing our data.
- 1:34
So I think if there's one thing I wanna convey with this whole talk is that you should really care about what you're training on, and you should care for it by giving it a nice format that does a lot of nice things for it.
- 1:46
I wanted to start just kinda broad strokes talking about how we think about pre-training and how we think of post-training. There's definitely a lot of overlap, but at least in terms of pre-training, you tend to th-think wider, right?
- 1:56
You wanna think about more like what domains you're training on. Are you thinking about books or more chat data? Um, and then you wanna also think about quantity, right?
- 2:05
How big is your model? How many tokens do you need? Compared to post-training where you're looking at very specific tasks, and maybe not just looking at the context of that task, but also how difficult is that math problem, how easy is that multiple choice problem?
- 2:19
So you kinda have to get much deeper and more granular in terms of the things that you understand about your data at scale.
- 2:26
In the middle, I, I guess I've grouped together some of my favorite problems right now that a lot of people are looking into. So ranging from data efficient learning, right, how do we reduce the amount of data we need to get good results from a similarly sized model?
- 2:40
How do we sample from data, right? Like, what kind of metrics do we need? And then how do we look at diversity, right? Measuring diversity is very difficult, and looking at some of the automated ways that we do that in industry and all the different papers that are out there.
- 2:56
So everyone loves a good hierarchy of needs. Uh, I think that for us, we always start with clean data and quickly go right up to evaluations. For us, we always start there because it's hard to measure anything without a compass, and since we're focused a lot on post-training nowadays, having systems for dataset management is becoming more and
- 3:15
more of a problem. So when we're thinking about mixtures, right, these collections of datasets, and usually you have different ways of how you're including them in your batches and in your training sets.
- 3:25
You wanna understand those collections, not just in terms of what dataset they are, you know, is this Wikipedia or is this some other thing, but also what's in there, right?
- 3:33
So it naturally rolls up into analytics. So we want token counts and understanding of length, and you might even want things that are more complicated, right? So you might be classifying your code data into not just, say, is this Python or is this Java, but also how difficult is it?
- 3:49
How many functions are in this problem? How many classes were you supposed to generate? And really having more and more analytics lets you understand your data more. I think more than anything, reading data has probably been the biggest win, so these are kind of just ways of automating things that we've learned from looking at data, looking at
- 4:06
outputs, looking at performance, and trying to understand what is going on.
- 4:11
Everything in kind of the top half, I guess, of this is more about using language models to improve language models, so things like synthetic data, things like quality scoring, things like dataset selection.
- 4:22
And dataset selection is probably the simplest and one of my favorites, right? You're just kind of looking at ways to match distributions for the behavior you want from your model and the data that you do have.
- 4:33
And so a lot of what we do is do retrieval or do clustering. You know, you can embed the web nowadays pretty quickly, and how do we pick the data that we like according to what kind of evaluations we're looking at?
- 4:45
Quality scoring is similarly simple. Like, we build a lot of classifiers in-house for a variety of things, and there's a lot of cool work around how people are actually doing this with just prompting classifications.
- 4:56
So you can do it even more simply than, say, having to go down the route of actually building a classifier and evaluating it and doing that whole loop.
- 5:05
And synthetics, given the way that we've structured our platform, is also super powerful for us because we have this ecosystem of big data tools like Spark and Trino alongside some GPU-backed services for doing prompting, for doing embedding, and for classifying things.
- 5:20
And so we can enrich our datasets. We can augment them. Um, you can generate quick examples of, say, preference pairs and try to explore a method, not at its peak of quality, right?
- 5:31
Synthetics are gonna have problems. But you can start getting signal for what types of data and what shape is that data and how can you kind of start looping in human labeling to make it even better.
- 5:42
So at the top, right, we use human labeling a lot for improving these classifiers, um, and we also wanna use them for rewriting synthetic data that maybe has issues or rewriting data that just has issues in itself.
- 5:54
And so all of this kinda comes together to motivate a lot of our, our platform tooling and, uh, we'll go on to the next slide to, to talk about kind of how we try to make all of this easy for researchers working in this domain
- 6:09
So I said at the beginning, accelerating research is a big part of this. I've included some beautiful YAML here that hopefully people can see that there's a SQL block over there.
- 6:17
And I think that this is pretty motivating in terms of how we materialize datasets. So if you've worked in machine learning at all, you know that usually you have a specific training format, maybe it's TFRecords, maybe it's JSON Lines, depending on where you're coming from.
- 6:33
And at least in my experience, it's one of the most error-prone components of training, right? I don't know what data this is. I'm training on it, I'm getting weird results.
- 6:41
So for our team, since we're doing so much iteration around data, making it part of your training job and separating concerns in terms of how your data's materialized and what your training job is doing is really, really nice for us.
- 6:55
And this is kind of where Lance started becoming a [REDACTED:sexual_orientation] deal, especially as we start thinking about multimodal and how the data volumes are much, much larger, and the problems that we're trying to solve become much more complicated.
- 7:07
So the materialization service aside, you know, it's kind of this nice interface that you send it some request, and it gives you some lists of, of files.
- 7:16
Um, really starts hitting the road when we think about data loading, which is its own problem in and of itself if, uh, especially once data volume becomes really large.
- 7:26
So Lance has this nice property that Chang will talk about a lot more that allows for quick random access, and it lets us shuffle data very cheaply, right? So it, it essentially lets you shuffle references to rows rather than shuffling the rows themselves, which allows you to save a lot of time in iteration speed.
- 7:43
And at the end of the day, for us, we just wanna watch the GPUs go brrr and the numbers go up. So I'll pass over to Chang, who can talk a lot more about the Lance format in detail.
- 7:52
Thanks. Cool. So you've heard from Noah about the importance of data in developing models. And so if data is critical, then it's also critical to have the right data infrastructure for the w- for your workloads.
- 8:08
Now, AI workloads tend to be a little bit different from your traditional data warehousing, OLAP, and analytics workloads i- in a couple of different ways. But let me give you just one motivating example.
- 8:20
If you think about a distributed training workload, typically it breaks down into three steps. You have a filter. You wanna select the right, uh, samples from your raw dataset.
- 8:32
Then you'll have a shuffle step, where you will then, uh, draw random rows, uh, from the filtered set, and then you'll stream-- Typically, if the dataset is large, you'll be streaming those observations, whether they're text or images or videos, from object storage into your GPUs.
- 8:52
So in that one workload, you needed fast scans to run s- the filter. You need r- fast random access to do the shuffling, and then you need to be able to deal with potentially very large binary data, large blobs, uh, to be able to quickly stream data in directly into your GPUs.
- 9:12
So these three properties are required often in one AI workloads from training to, uh, search and retrieval, but existing data formats and data infrastructure is good for at most two, but, but often just one of the three.
- 9:28
And so this is what I'm calling the, the new CAP theorem for AI data, um, and that's the motivation for us for designing, uh, Lance format, uh, around which we've built LanceDB.
- 9:41
So, uh, this problem is of course exacerbated by scale of AI data, and especially multimodal data. So if you look at tabular data from the past, one row of tabular data j- just scalar, simple scalar columns, on average is about a hundred and fifty bytes per row.
- 10:01
Uh, if you add embeddings to that, that gets about twenty, twenty, twenty-five times larger, depending on the number of dimensions. If you add images, uh, that's another twenty times, and if you add videos, that gets pretty astronomical, and that's one single row.
- 10:17
Um, and with generative AI, data it isn't limited by the speed at which, you know, manual human interaction can generate observations. Uh, n- new, new rows of data is being generated at thousands of tokens per second.
- 10:32
So scale often blows, blows up. Um, in the past, 'cause I've been in data for a long time, if you were in the tens of terabytes, you were a fairly large company, and I think these days, uh, if you're working in generative AI, it's not unheard of for, you know, uh, ten-person, twenty-person teams to be managing, like,
- 10:52
tens of terabytes to even pe- uh, petabytes o- of data.
- 10:57
Um, so what does Lance format do to solve these problems? Well, so Lance format, first it's a columnar file format, so like Parquet, but-- or, uh, optimized for AI.
- 11:09
So it gives you the ability to do fast scans like Parquet. It supports fast lookups, unlike Parquet. And, uh, we've actually gotten rid of a big limiting factor in Parquet called row groups, uh, and so that we can allow you to store blobs in-line.
- 11:27
Lance format is also a lightweight table format, so as you a- add data, uh, it's automatically versioned. You can also add additional columns without having to copy the original dataset.
- 11:40
So it makes it a lot easier if you're working with large, uh, multimodal datasets to, uh, add experimental features and then roll, roll them back later on. Um, and we'll call this a zero-copy schema evolution.
- 11:54
And then finally, of course, it supports time travel, so that oftentimes if you make a mistake or there's an error or there's bad data, it's instantaneous to roll back to a previously known good version so that it doesn't corrupt, uh, down- downstream, um, model training processes.
- 12:12
And the third aspect of Lance format that's really interesting is indexing extensions. So in Parquet, there are indices, but, uh, the indices can quickly tell you which rows you need.
- 12:24
But with Parquet, because it doesn't support random access, even if you know which rows you need to fetch, it's really slow to fetch those rows, um, and, and not so with Lance.
- 12:33
So with, uh, with Lance, we've added indexing extensions for embeddings, so you can do, you know, essentially billion-scale vector search directly off of S3. Um, you can-- we can have scalar indices to make filtering metadata columns really quickly, uh, and then, uh, full-text search indices to do keyword or fuzzy search, uh, and directly from, uh, from
- 12:58
your S3 dataset. Uh, you-- and you don't really need that, uh, Elasticsearch cluster anymore. So what Lance gives you is the ability to have a single table for many, many different workloads.
- 13:12
So, uh, if you have metadata columns or time series columns, you can run SQL. So you can plug Lance directly into, say, um, DuckDB or, uh, Trino, uh, or Spark, and you can run SQL on that.
- 13:27
Uh, you-- if you're storing large blobs and tensors like te- uh, the videos or text or images, you can plug your Lance data, the same table, into PyTorch training.
- 13:39
Um, and if you have embedding vectors, you can use the embedded, uh, uh, vector index to do similarity search.
- 13:48
And so this makes it a lot easier, uh, for a full AI workflow from, um, analyzing and exploring your dataset to searching and retrieving throughout your dataset to, uh, fine-tuning and training, uh, your model.
- 14:05
Around this format, we built LanceDB, um, uh, vector database and the, more generally, database for multimodal AI. So, uh, one big feature is a distributed vector search, so billion, uh, search through billions of vectors at low latency and very high QPS with order of magnitude less infra than other vector databases.
- 14:28
And it provides data infrastructure for all of your multimodal data needs. When we talk about multimodal, we often think narrowly about just image generation or video generation. But when you look at the data, multimodal, I think, has many different meanings.
- 14:42
One, of course, is the data. The data can be, uh, multimodal. So un-unlike traditional tabular data, we can store features, um, and then audio waveforms, images, and all that.
- 14:52
That's what we're familiar with that already. Um, and of course, vectors, and, and vector is a vector, so whether they're, they're image, uh, embeddings or, or text embeddings. Now, the workload can also be multimodal.
- 15:04
Um, so, you know, not just running OLAP SQL, but you can run vector search, you can run full-text search, uh, and filtering, and then, um, uh, and other, uh, sort of data frame and SQL workloads.
- 15:18
And then finally, the use case and the scenario can also be multimodal. So, uh, operational scenarios where you're in a production service for, uh, RAG or search and retrieval and personalization, or Lance can be used in training, uh, or it can be part of your data lake to analyze and explore all that multimodal data that you have.
- 15:45
Yeah. So I think that from at least my team's experience and a lot of what Chang is describing, we just think that speed is probably our, our best bet in terms of strategy.
- 15:55
And a lot of the tools that we've worked with really slow down under load, under new multimodal needs, and we're looking to develop out what the future for those data systems looks like.
- 16:07
So thanks so much for listening to our talk. Yeah. [outro music]