# From Tokens to Cells: Foundation Models for Single-Cell Biology - Akram Baharlouei, Altos Labs

Akram Baharlouei

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/-561cZmir5Q-from-tokens-cells-foundation-models-single-cell) · [Video](https://www.youtube.com/watch?v=-561cZmir5Q)

## From Tokens to Cells: Foundation Models for Single-Cell Biology

Reprogramming a cell makes its state a modeling problem: what can we measure, what does sequencing miss, and can foundation models predict the full range of cellular responses?

Basic familiarity with machine-learning training and vector representations is helpful; no biology background is required.

### What would it take to restore a cell’s youthful function?

[0:12](https://www.youtube.com/watch?v=-561cZmir5Q&t=12s)

What would it take to restore a cell’s youthful function without changing what kind of cell it is? That question motivates cellular rejuvenation and the mission Akram Baharlouei describes at Altos Labs: restoring cell health and resilience to address disease and disability. His perspective is that of a machine-learning engineer entering biology, with an emphasis on the engineering challenges between measuring cells and training useful foundation models.

The starting example is reprogramming. In 2006, Shinya Yamanaka discovered that overexpressing four transcription factors could push a differentiated cell toward an embryonic stem-cell-like state. In the talk’s skin-cell example, the intervention changes cell identity: a specialized cell becomes an **induced pluripotent stem cell**, or iPSC. The discovery, recognized with a Nobel Prize in 2012, opened possibilities for regenerative medicine, including producing cells needed to repair tissues or organs.

![A complete turquoise cell with a central nucleus beneath the label “Induced pluripotent stem cell (iPSC).”](https://ai.engineer/api/talk-essay-media/7a4b91ecfcc65ec2d8f5dd24c5bd1321ae1544455086b975d147dccd565ccbdf)

The reprogramming animation reaches an induced pluripotent stem cell (iPSC).

**Partial reprogramming** pursues a different endpoint: restoring youthful function while retaining cell identity. A rejuvenated skin cell should still be a skin cell. Baharlouei presents this as a therapeutic ambition, including the possibility of an mRNA medicine, rather than an established treatment. He tentatively points to an OSK-based human-testing milestone in 2026, roughly two decades after the original discovery; entering human testing is distinct from demonstrating rejuvenation in patients.

### From cell models to the drug-development pipeline

[3:11](https://www.youtube.com/watch?v=-561cZmir5Q&t=191s)

Reprogramming is one reason to understand cellular state. A broader ambition is to build a unified model of a cell, then use it as a foundation for models of tissues, organs, and eventually humans. The [Human Cell Atlas](https://www.humancellatlas.org/learn-more/about-the-human-cell-atlas/) is an ongoing effort to map human cell types, not a completed census of every human cell. Virtual cells, virtual tissues, virtual humans, and digital twins name related ambitions at different biological scales: models that could help explain how organisms work and support treatment development.

The practical pressure comes from drug development. Baharlouei contrasts the rapid improvement associated with Moore’s Law with what he describes as declining drug-development productivity. Failure is routine: he offers a rough success estimate of five percent or less, without specifying a denominator or development stage. He describes a research-to-clinic pipeline that can take ten years and cost billions of dollars. These are broad estimates motivating the problem, not measurements of a defined cohort.

**Faster discovery does not automatically make the whole pipeline fast.** Research, preclinical work, and clinical trials impose successive constraints. Better protein design might save time in early research while leaving later stages largely unchanged. The attraction of virtual biological models is therefore wider than generating promising molecules: the goal is to improve decisions across the development pipeline, where downstream bottlenecks can otherwise absorb gains made upstream.

### What can we measure inside a cell?

[5:56](https://www.youtube.com/watch?v=-561cZmir5Q&t=356s)

A cell contains much more information than any single measurement captures. Its genome supplies DNA sequence, which is broadly shared across cells; its transcriptome describes which genes are expressed and in what quantities at a particular time. For machine learning, single-cell RNA sequencing produces a familiar structure: a **cell-by-gene matrix**, with cells as rows, genes as columns, and measured expression quantities as entries. The talk places the gene dimension at roughly 20,000.

| Measurement | What it describes | Main distinction |
| --- | --- | --- |
| Genome | DNA sequence | Broadly shared across cells |
| Transcriptome | Gene-expression profile | Varies with cellular state |
| Proteome | Proteins present in a cell | Functional machinery; difficult to measure at scale |
| Morphology | Cell shape and structure | Accessible through microscopy |
| Spatial information | Location within tissue | Preserves the cell’s surroundings |

Proteins carry out many cellular functions, making proteomics especially valuable, but Baharlouei describes its measurement as difficult and comparatively low throughput. Microscopy adds shape and structure, while spatial measurements help locate an individual cell within tissue. These modalities describe different aspects of the same biological system.

RNA-seq dominates much single-cell foundation-model training because it is comparatively easy to measure and scale, helped by PCR-based amplification. Baharlouei describes collections containing tens of millions of cells, says he has heard of a 500-million-cell collection, and mentions a billion-cell project. Those larger examples are unnamed; their scale should not be read as evidence that the samples are all independent or experimentally comparable. Even an enormous RNA-seq corpus remains a measurement of gene expression, so progress in other modalities is still necessary for a fuller model of cell biology.

### A billion snapshots are still snapshots

[8:30](https://www.youtube.com/watch?v=-561cZmir5Q&t=510s)

Suppose a training collection did contain a billion RNA-seq samples. Would that be sufficient? The difficulty is that nominally identical cells need not produce identical measurements. Cells grow, change their molecular activity, and divide. Some activity happens in bursts. Sequencing captures one state during that process, not the trajectory that produced it: a snapshot from a biological movie. Increasing the number of snapshots does not by itself reveal the temporal history of each cell.

![Under “scRNA-seq captures a snapshot,” colored cells are labeled G1, S, M, and C beneath the title “A cell is a movie — we only capture snapshots.”](https://ai.engineer/api/talk-essay-media/05fcdb62eb61089efbd233cbf6c03e24dec04c614b95c25f88e495fb78c3d619)

Single-cell sequencing captures cells at different points in their cycle.

Biological variation is only part of the problem. Different laboratories and machines can produce different measurements, introducing **technical variation** alongside real differences in cellular state. A useful model must learn from both without simply treating every difference as biology. The data’s high dimensionality and many possible states prompt a brief quantum-computing aside, but Baharlouei presents that as speculation, with no demonstrated advantage. The immediate question is what current AI methods can extract from the measurements already available.

### Cells as sentences, genes as tokens

[10:30](https://www.youtube.com/watch?v=-561cZmir5Q&t=630s)

**scGPT** and **Geneformer** bring transformer-based representation learning to single-cell data. The analogy is intuitive: treat a cell like a sentence and genes like tokens. Baharlouei explains the family through **BERT**-style masked prediction: hide part of a cell’s gene information and train a model to recover it from the remaining context. This is a shared conceptual introduction, not a claim that scGPT and Geneformer use identical encodings or training objectives.

For expression-value reconstruction, the cell-by-gene matrix makes the learning problem concrete. The following Python example hides two entries in a small expression vector and keeps their original values as targets. The mask is separate from the filled input, so a hidden measurement is distinguishable from an observed zero.

```python
import numpy as np

genes = np.array(["GENE_A", "GENE_B", "GENE_C", "GENE_D"])
expression = np.array([12.0, 0.0, 7.0, 3.0])
masked = np.array([False, True, True, False])

model_input = expression.copy()
model_input[masked] = 0.0

target_genes = genes[masked]
target_values = expression[masked]

print(model_input)    # [12.  0.  0.  3.]
print(target_genes)   # ['GENE_B' 'GENE_C']
print(target_values)  # [0. 7.]
```

The training signal asks the model to recover hidden information using relationships among genes. Bidirectional attention provides a way to use the surrounding gene context rather than predicting each entry independently.

The resulting cell representation can support **cell-type prediction** or **perturbation-response modeling**: predicting how a cell responds to an intervention. In the architecture pattern described here, the model compresses the input into a latent vector. A classification head uses that representation to assign a label; a decoder uses it to generate a predicted output. The important question is whether the representation retains the information needed by the downstream task.

### Training expense does not guarantee useful representations

[12:26](https://www.youtube.com/watch?v=-561cZmir5Q&t=746s)

Compression can discard information that matters for prediction. Baharlouei offers this as an explanation for disappointing downstream results, rather than establishing it as the universal cause. He reports that simple linear models sometimes match or outperform expensive single-cell foundation models. That comparison makes the baseline essential: a complex representation is valuable only if it improves the task being evaluated.

The benchmarking work discussed next covers two settings: imaging together with RNA-seq, and perturbation-response modeling. Baharlouei describes comprehensive evaluations whose results echo the broader concern that costly training has not produced gains comparable to those seen in language and imaging. The claim is qualitative here; it does not establish a single ranking across every model, dataset, and biological task.

### Predicting a distribution rather than its mean

[13:44](https://www.youtube.com/watch?v=-561cZmir5Q&t=824s)

**Flow matching** offers another way to formulate the prediction problem. Begin with samples of Gaussian noise and learn a transformation toward the distribution of cellular measurements. The goal is to reproduce a population’s range of responses, rather than reducing that population to a single representative prediction. Baharlouei introduces [**PRiMeFlow**](https://arxiv.org/abs/2604.13986), available as an arXiv preprint, as a promising example of this approach.

The distinction matters when cells respond heterogeneously. A prediction near the average can fail to represent the different states actually present in a population. Baharlouei contrasts distribution matching with approaches organized around a compressed latent representation and reports favorable results for flow matching. Those results should be understood in the perturbation-response setting discussed here, not as universal superiority over transformer, autoregressive, or autoencoder architectures.

In the displayed comparison, Baharlouei identifies the green points as ground truth. He describes PRiMeFlow as following their distribution, while the autoencoder comparator concentrates predictions near the mean. The accompanying **maximum mean discrepancy**, or **MMD**, score addresses distributional agreement. The useful distinction is between reproducing an average response and reproducing the spread of observed responses; the talk supplies no numerical values here to quantify the difference.

### Better measurements must accompany scale

[15:11](https://www.youtube.com/watch?v=-561cZmir5Q&t=911s)

The ambitions behind single-cell modeling—cellular rejuvenation, virtual biological systems, and a shorter drug-development cycle—depend on measurements that capture the relevant biology. RNA-seq is both useful and comparatively abundant, but **more data and better measurements are separate requirements**. Scaling the existing collection process alone does not ensure that the resulting corpus represents a living organism more faithfully.

Flow matching is the promising direction Baharlouei leaves open, especially for representing heterogeneous perturbation responses. The harder goal is to scale models that work on data they have not seen during training. That requires larger collections together with improved quality and measurement methods. He closes by pointing to [scGeneScope](https://github.com/altoslabs/scGeneScope) and [PerturBench](https://github.com/altoslabs/perturbench), which he identifies as prior-year NeurIPS work, alongside PRiMeFlow: a progression from testing what current representations preserve to modeling the cellular variation those representations need to capture.

## Resources

- [PRiMeFlow: modeling heterogeneous perturbation responses](https://arxiv.org/html/2604.13986v1) — A flow-matching model operating directly in gene-expression space, with perturbation-response benchmarks and distributional evaluation.
- [scGeneScope](https://github.com/altoslabs/scGeneScope) — Code, data-access instructions and training workflows for a perturbationally paired imaging and transcriptomics benchmark.
- [PerturBench](https://github.com/altoslabs/perturbench) — Tools for evaluating cellular perturbation predictions with multiple aggregation methods and metrics.
- [About the Human Cell Atlas](https://www.humancellatlas.org/learn-more/about-the-human-cell-atlas/) — The consortium's overview of its ongoing effort to map human cell types across tissues and the lifespan.
- [ER-100 receives clearance to enter human trials](https://www.lifebiosciences.com/life-biosciences-announces-fda-clearance-of-ind-application-for-er-100-in-optic-neuropathies/) — A sponsor announcement describing an OSK-based investigational therapy and its planned Phase 1 evaluation for optic neuropathies.

## Slides

### From Tokens to Cells

Akram Baharlouei introduces the engineering challenges of foundation models for single-cell biology from her perspective as a machine learning engineer without a biology background.

[0:00](https://www.youtube.com/watch?v=-561cZmir5Q&t=0s) [Title slide for a talk on engineering challenges of foundation models for single-cell biology.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4c37611fd3b0dddac074af93fbe5e1fdef1dc0791fb0cf9ef18519335b86c5f.webp)

AI ENGINEER TALK

From Tokens to Cells
The engineering challenges of foundation models for single-cell biology

Akram Baharlouei
Altos Labs

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#1
Ask Gemini
akram baharlouei

### Outline

The talk moves from why single cells matter through measurement and heterogeneity to cell modeling and takeaways.

[0:30](https://www.youtube.com/watch?v=-561cZmir5Q&t=30s) [Outline covering single cells, measurement, heterogeneity, modeling cells, and takeaways.](https://assets.ai.engineer/sites/aiecode2025/sha256/a18e189bb8c6bc17c3ebe724b25e01f7f1e3c1e36ee666ed27cb714b9ab2b08d.webp)

OUTLINE

01 Why single cells
02 What to measure
03 Why it’s hard: heterogeneity
04 Modeling cells
05 Takeaways

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#2
Ask Gemini
akram baharlouei

### Yamanaka factors reset a cell to a stem-like state

The animation shows four factors entering an aged cell and docking on nuclear DNA, alongside an old-to-young epigenetic age scale. Baharlouei describes partial reprogramming as changing cell age without changing cell type.

[1:21](https://www.youtube.com/watch?v=-561cZmir5Q&t=81s) [A cell visualization shows transcription factors entering, with an epigenetic age indicator and playback controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/50b033f2f27d591190110cc6cbf6b3f2ba23b56c1a409b2ab8db796d6425755b.webp)

CELLULAR REPROGRAMMING
Yamanaka factors reset a cell to a stem-like state
OSKM · ALTOS

Transcription factors entering

EPIGENETIC AGE
OLD — YOUNG

PAUSE
REPLAY
REC

Oct4 · Sox2 · Klf4 · c-Myc

Oct4 · Sox2 · Klf4 · c-Myc are transcription-factor proteins. They enter an aged cell, dock onto DNA in the nucleus, and rewind its epigenetic age toward an induced pluripotent stem cell.
Auto-orbit; click the scene to play / pause.

Interface text:
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#4
Ask Gemini
akram baharlouei

[2:31](https://www.youtube.com/watch?v=-561cZmir5Q&t=151s) [An animated cell with transcription factors entering and an epigenetic age indicator.](https://assets.ai.engineer/sites/aiecode2025/sha256/b818d598d65391d5d97a425c22050e49d2027dec492edb544eb92e55b9675c15.webp)

CELLULAR REPROGRAMMING
Yamanaka factors reset a cell to a stem-like state
OSKM · ALTOS

Transcription factors entering

EPIGENETIC AGE
OLD — YOUNG

PAUSE
REPLAY
REC

Oct4 · Sox2 · Klf4 · c-Myc

Oct4 · Sox2 · Klf4 · c-Myc are transcription-factor proteins. They enter an aged cell, dock onto DNA in the nucleus, and rewind its epigenetic age toward an induced pluripotent stem cell.
Auto-orbit; click the scene to play / pause.

Interface text:
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#4
Ask Gemini
akram baharlouei

### Build the virtual human like a pyramid

Starting with genes, transcripts and proteins in a cell, the slide builds toward tissue, organ and body simulation. Baharlouei frames a virtual human as a long-term goal.

[3:10](https://www.youtube.com/watch?v=-561cZmir5Q&t=190s) [A sequence of four boxes progresses from cell to tissue, organ, and body.](https://assets.ai.engineer/sites/aiecode2025/sha256/72717eb53d00011c0c861a680e090c338d0b7bb88b2d9264094709696bbdc35d.webp)

01 · WHY SINGLE CELLS

Build the virtual human like a pyramid

FOUNDATION
Cell
Genes, transcripts, proteins

LAYER
Tissue
Cells talking to neighbors

PEAK
Organ
Tissues strung together

SCALE
Body
A simulated population

Master the simulation of one unit and you gain the code to simulate the organism. (Sarah Teichmann & the Human Cell Atlas.)

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#6
Ask Gemini
akram baharlouei

### AI at every stage of drug development

The slide maps AI approaches across four drug-development stages, from protein engineering to digital twins. Baharlouei’s goal is to shorten the whole pipeline, not just accelerate early research.

[4:57](https://www.youtube.com/watch?v=-561cZmir5Q&t=297s) [Four columns trace research, preclinical studies, clinical trials, and review and approval, with AI applications beneath each stage.](https://assets.ai.engineer/sites/aiecode2025/sha256/73a43cd4c4fd84cd6dc5fb7164f3aaa60a8cd1bb1802f258ddaa2082341dde92.webp)

AI at every stage of drug development

1. Research & Development
3–6 years
Target identification
Compound screening
Lead optimization

AI REPLACES
Protein engineering models
Scientific reasoning agents

2. Preclinical Studies
1–2 years
In vitro studies
In vivo animal models
Toxicity testing

AI REPLACES
Virtual cells
Virtual tissues & organs
In silico toxicity

3. Clinical Trials
4–7 years
Phase I, II & III trials
Dosage & safety monitoring
Patient recruitment

AI REPLACES
Virtual humans
In silico trials
Synthetic patient cohorts

4. Review & Approval
1–2 years
Safety & efficacy review
Regulatory approval
Post-market surveillance

AI REPLACES
Digital twins
Adverse event prediction
Continuous monitoring

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#9
Ask Gemini
akram baharlouei

### Many ways to read a cell

Different measurements capture different aspects of one cell, from DNA and chromatin to proteins, shape and location. The slide highlights scRNA-seq, which represents each cell as a roughly 20,000-gene transcriptome vector.

[5:54](https://www.youtube.com/watch?v=-561cZmir5Q&t=354s) [A cell diagram and a list compare transcriptomics, genomics, epigenomics, proteomics, spatial measurements, morphology, and imaging.](https://assets.ai.engineer/sites/aiecode2025/sha256/70541a6374db5f1b2b318cebef0f72fd9fb4dd58d7ee8108a8dad9fd0a5d8158.webp)

02 / THE DATA

Many ways to read a cell
One cell — many measurement modalities, each capturing different biology.

Morphology
cell shape & structure

Epigenomics
chromatin access

Genomics
DNA sequence

scRNA-seq ★
transcriptome

Proteomics
protein output

Spatial
location in tissue

scRNA-SEQ — OUR FOCUS
Transcriptome
~20,000-gene vector per cell · high throughput · millions of cells

GENOMICS
The DNA code
Static sequence · same in every cell · genome-wide variants

EPIGENOMICS
Chromatin accessibility
Which DNA regions are open · histone modifications

PROTEOMICS
Protein output
Actual functional molecules · low throughput · expensive

SPATIAL
Location in tissue
Gene expression + tissue coordinates · cell neighborhoods

MORPHOLOGY
Cell shape & structure
Size, shape, organelle layout · light & electron microscopy

IMAGING
Fluorescence & microscopy
Protein localization, live-cell imaging · high-content screening

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#11
Ask Gemini
akram baharlouei

### Many ways to read a cell

Each measurement captures a different aspect of a cell. The talk focuses on scRNA-seq: it scales to millions of cells, while proteomics measures functional molecules at lower throughput.

[6:01](https://www.youtube.com/watch?v=-561cZmir5Q&t=361s) [A cell diagram and cards compare transcriptomics, genomics, epigenomics, proteomics, spatial measurements, morphology, and imaging.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9c5cf064bd4ae903d302cd7442ada40b1ca9e1ea7cc0a70a50a3e6df31f9e8a.webp)

Many ways to read a cell
One cell — many measurement modalities, each capturing different biology.

Morphology
cell shape & structure

Epigenomics
chromatin access

Genomics
DNA sequence

scRNA-seq ★
transcriptome

Proteomics
protein output

Spatial
location in tissue

scRNA-SEQ  OUR FOCUS
Transcriptome
~20,000-gene vector per cell · high throughput · millions of cells

GENOMICS
The DNA code
Static sequence · same in every cell · genome-wide variants

EPIGENOMICS
Chromatin accessibility
Which DNA regions are open · histone modifications

PROTEOMICS
Protein output
Actual functional molecules · low throughput · expensive

SPATIAL
Location in tissue
Gene expression + tissue coordinates · cell neighborhoods

MORPHOLOGY
Cell shape & structure
Size, shape, organelle layout · light & electron microscopy

IMAGING
Fluorescence & microscopy
Protein localization, live-cell imaging · high-content screening

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#11
Ask Gemini
akram baharlouei

### Why it’s hard: heterogeneity

Even identical cells can produce different readings, and some variation is not signal. That makes single-cell RNA-seq data difficult to use for training foundation models.

[8:28](https://www.youtube.com/watch?v=-561cZmir5Q&t=508s) [Section divider introducing heterogeneity in cell measurements.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9e4bfbcf812067b56b87674f9bfa7bd30aa3853607c55ca1cea4bd3383a7455.webp)

SECTION 03
Why it’s hard: heterogeneity
Two “identical” cells rarely read the same — and not all of it is signal.

Interface text:
localhost:8765/From Tokens
Ask Gemini
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#12
akram baharlouei

### A cell is a movie — we only capture snapshots

Cells grow, copy DNA and divide into two daughter cells. scRNA-seq destroys each cell to read its mRNA at one moment, capturing cells at different cycle positions and contributing to noise and heterogeneity.

[8:56](https://www.youtube.com/watch?v=-561cZmir5Q&t=536s) [A cell-cycle animation shows division into two daughter cells.](https://assets.ai.engineer/sites/aiecode2025/sha256/86308bd0f0f6261109ef248bf9b6aaad5739cbe497fa7715fdab78a619702984.webp)

WHY SINGLE-CELL DATA IS NOISY
A cell is a movie — we only capture snapshots

CYTOKINESIS · [illegible]

C
DIVIDE
cleavage furrow pinches cell in two

2 daughter cells

04 · C
DIVIDE

A cell continuously cycles: grow → copy DNA → split chromosomes → divide. scRNA-seq destroys the cell and reads its mRNA at one frozen moment — giving thousands of disconnected snapshots at random cycle positions. That is the core source of noise and heterogeneity.

Interface text:
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#13
Ask Gemini
akram baharlouei
PAUSE
REPLAY
REC

### A cell is a movie — we only capture snapshots

Cells in the same tissue can be captured at different points in their cycles, with different gene expression. The animation shows how scRNA-seq turns a changing population into disconnected snapshots.

[9:10](https://www.youtube.com/watch?v=-561cZmir5Q&t=550s) [A population diagram shows cells captured at different stages of the cell cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/552445019a33e2748cd3b9b64465c78102a36c152688b761829dc77850185480.webp)

WHY SINGLE-CELL DATA IS NOISY
A cell is a movie — we only capture snapshots

POPULATION · SCRNA-SEQ [illegible]

scRNA-seq captures a snapshot
each cell frozen at a different moment in its cycle
same tissue · different gene expression = heterogeneity

G1  S  G1  [illegible]  C  G1
S  G1  M  G1  S
M  G1  C  G1  S
G1  M  S  C

A cell continuously cycles: grow → copy DNA → split chromosomes → divide. scRNA-seq destroys the cell and reads its mRNA at one frozen moment — giving thousands of disconnected snapshots at random cycle positions. That is the core source of noise and heterogeneity.

Interface text:
PAUSE
REPLAY
REC
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#13
Ask Gemini
akram baharlouei

### Technical: variation that isn’t biology

Differences in labs, machines, sequencing depth and protocols can change cell measurements. Dropout and ambient RNA add further confounds that can look like biological signal.

[9:36](https://www.youtube.com/watch?v=-561cZmir5Q&t=576s) [A list describes batch effects, dropout, sequencing depth, ambient RNA, and platform differences.](https://assets.ai.engineer/sites/aiecode2025/sha256/04a0e77f1a31d8ad03da58dbe6035620c72b300f56e790e00849c2eb06a72d7d.webp)

03 · HETEROGENEITY

Technical: variation that isn’t biology

batch-effect — Two labs, two machines, two days — different numbers for the same biology.
dropout — Expressed genes read as zero — absence you can't trust.
depth — Sequencing depth varies cell to cell, rescaling every count.
ambient-RNA — Free-floating transcripts contaminate every droplet.
platform — Different chemistries and protocols aren't directly comparable.

The trap: confounds look exactly like signal. Disentangling them is half the modeling problem.

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#14
Ask Gemini
akram baharlouei

### Modeling cells

This section examines foundation models for cell data, moving from transformers to approaches that fit the data.

[9:51](https://www.youtube.com/watch?v=-561cZmir5Q&t=591s) [Section divider introducing models for cell data.](https://assets.ai.engineer/sites/aiecode2025/sha256/2966bb0bc4ce6bc36128742053de29a856883cd6489c5ba36b0a2134426b3a34.webp)

SECTION 04
Modeling cells
From the obvious choice — transformers — to what actually fits the data.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#15
Ask Gemini
akram baharlouei

### Quantum computing and biology

Quantum state spaces may fit the complexity of gene co-expression, but current hardware is noisy and small-scale. The slide says no quantum algorithm has shown a real advantage on biological data.

[9:57](https://www.youtube.com/watch?v=-561cZmir5Q&t=597s) [Slide contrasts the intuition for quantum computing in biology with current hardware and algorithm limitations.](https://assets.ai.engineer/sites/aiecode2025/sha256/4014cc6a2a52b83a57ef6824c87463ecacf4709f54cc31010f2b12adfabd1141.webp)

04 · MODELING CELLS · AN OPEN QUESTION

Quantum computing might be a better fit for biology in the future

A cell’s state lives in a ~20,000-dimensional gene space. Classical computers search it sequentially. Quantum computers explore many states at once — which might be a natural match. Nobody knows yet.

THE INTUITION
Quantum state spaces scale exponentially with dimension — on paper, that matches the combinatorial complexity of gene co-expression. Molecular simulation is also a natural quantum application.

WHERE WE ACTUALLY ARE
Hardware is noisy, small-scale, and nowhere near the qubit counts needed. No quantum algorithm has shown a real advantage on biological data.

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#16
Ask Gemini
akram baharlouei

### Treating genes like tokens

scGPT and Geneformer adapt language-model training to cells. The slide flags unordered genes, lost expression magnitudes, and sparse zeros as limits of treating genes as tokens.

[10:26](https://www.youtube.com/watch?v=-561cZmir5Q&t=626s) [Slide describes scGPT and Geneformer and lists challenges with treating gene expression as tokens.](https://assets.ai.engineer/sites/aiecode2025/sha256/041bcd28e72b4c6918100913c852e0c640f50f68389eeaef79d54318300ce314.webp)

04 · MODELING CELLS

First instinct: treat genes like tokens

WHAT WE TRIED
scGPT, Geneformer & friends
Rank or tokenize expression, run the language-model playbook, self-supervise on millions of cells with masked / next-gene prediction.

TRAINING DATA
scGPT 33M human cells · CellxGene Census · 51 tissues
Geneformer 29.9M cells · 100+ datasets · 30 organs

WHY IT STALLS
No order. Genes aren't a sequence — position is meaningless.
Continuous, not categorical. Expression is a magnitude; tokenizing discards it.
Mostly zeros. A ~20k-dim, sparse vector wastes the objective on absence.

The token objective optimizes the wrong thing — it discards the magnitudes that carry the biology. → sc-gene-scope

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#17
Ask Gemini
akram baharlouei

### scGPT: Cell–gene expression matrix

The matrix highlights one cell’s expression values across genes. scGPT masks some values and trains the model to reconstruct them.

[10:58](https://www.youtube.com/watch?v=-561cZmir5Q&t=658s) [A three-dimensional cell-by-gene matrix highlights expression values in one row.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed70951187ee67d6114198e397fecfcce0a03d241429858afcfead1449e49051.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE MA[illegible]

STEP
Cell × Gene Expression Matrix

cells
genes
cells as rows · genes as columns · values = log1p(counts)

PAUSE
REPLAY
REC
expressed
[MASK] / predicted
attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values — the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei

### scGPT: Each gene is a token

scGPT represents each gene as a token combining its identity and expression value. The demo highlights one cell’s row in the cell-by-gene matrix.

[11:11](https://www.youtube.com/watch?v=-561cZmir5Q&t=671s) [Gene names label columns above a highlighted row of expression values.](https://assets.ai.engineer/sites/aiecode2025/sha256/50867930b40e1b8643dbb9f770525f509da939ed475b19b01fae6ad48e2fc233.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE MA[illegible]

STEP
Each gene = a token

CD3E  CD19  GAPDH  TP53  IL7R  MS4A1  GNLY  CD14  LYZ  NKG7  FCER1A  ACTB  CD4  CD8A

#5 →
cells
[illegible]

PAUSE  REPLAY  REC
expressed  [MASK] / predicted  attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values — the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei

### scGPT: Masked Gene Modeling

The demo masks about 15% of gene expression values in a cell. Question marks replace selected values, which the model is trained to predict from the other genes.

[11:13](https://www.youtube.com/watch?v=-561cZmir5Q&t=673s) [The matrix highlights several gene expression values in gold under a heading about masking approximately 15 percent of values.](https://assets.ai.engineer/sites/aiecode2025/sha256/e56894e8eeeb50674b189bd3aac47c13860566998161552c280f6de21dac1359.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE MA[illegible]

STEP
Mask ~15% of gene expression values

CD3E  CD19  GAPDH  TP53  IL7R  MS4A1  GNLY  CD14  LYZ  NKG7  FCER1A  ACTB  CD4  CD8A

expressed  [MASK] / predicted  attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values — the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei
PAUSE  REPLAY  REC

[11:15](https://www.youtube.com/watch?v=-561cZmir5Q&t=675s) [A gene expression matrix highlights masked values with question marks.](https://assets.ai.engineer/sites/aiecode2025/sha256/a4150b4907e3171e0b02d773faaa06c36be52de972eb8079d84e5ca3833052df.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE MA[illegible]

STEP
Mask ~15% of gene expression values

CD3E  CD19  GAPDH  TP53  IL7R  MS4A1  GNLY  CD14  LYZ  NKG7  FCER1A  ACTB  CD4  CD8A
[MASK]  [MASK]  [M[illegible]
?  ?  ?

expressed  [MASK] / predicted  attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values — the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei
PAUSE  REPLAY  REC

### scGPT: Reconstructing Masked Values

The demo reconstructs masked entries in a cell-by-gene expression matrix, using the other genes to predict the missing values.

[11:26](https://www.youtube.com/watch?v=-561cZmir5Q&t=686s) [The gene expression matrix shows reconstructed values as yellow bars.](https://assets.ai.engineer/sites/aiecode2025/sha256/308349ee5b3bf48d7bc045121969c038456cc5268a7a11113d69dbbf8fc5b7d5.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE [illegible]

STEP
Reconstruct masked expression values

[illegible]3E  CD19  TP53  IL7R  MS4A1  GNLY  LYZ  NKG7  FCER1A  ACTB  CD4  CD8A
[MASK]  [MASK]

[illegible]

expressed  [MASK] / predicted  attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values—the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei
PAUSE  REPLAY  REC

### scGPT: Masked Gene Modeling

The demo highlights one cell’s expression values across a cell-by-gene matrix. Each gene becomes a token; scGPT masks values and uses bidirectional attention to reconstruct them.

[11:28](https://www.youtube.com/watch?v=-561cZmir5Q&t=688s) [A perspective grid shows gene expression values for cells, with one row highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/d374d93585641af0a31d3edf00af86afeed877951a4d3b99f9744b539a9966f5.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE [illegible]

STEP
Cell × Gene Expression Matrix

cells
Gene →
cells as rows · genes as columns · values = [illegible]

PAUSE
REPLAY
REC
expressed
[MASK] / predicted
attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values—the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei

[11:40](https://www.youtube.com/watch?v=-561cZmir5Q&t=700s) [Labeled genes appear above expression bars in a matrix.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb2ee9d940cd7c931a13296fbea0b0de0fee0e4fee51f5b08fc7d2b45d2e0a38.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE [illegible]

STEP
Each gene = a token

CD19 GAPDH TP53 IL7R MS4A1 GNLY CD14 LYZ NKG7 FCER1A ACTB CD4 CD8A
cells
Gene →
[illegible] expression value = one input token

PAUSE
REPLAY
REC
expressed
[MASK] / predicted
attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values—the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei

### Mask gene expression values

The demo masks ~15% of gene expression values in a cell’s row. scGPT learns to reconstruct the masked values using the unmasked genes.

[11:43](https://www.youtube.com/watch?v=-561cZmir5Q&t=703s) [Several expression bars are highlighted as the demo masks approximately 15% of values.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d558d25ab35f3186dd7bbdfe99ec7f3ab3637c665ccb12a02ee059fac30abc2.webp)

TRANSFORMER-BASED MODELS
scGPT: Masked Gene Modeling
BERT-STYLE · CELL × GENE MA[illegible]

STEP
Mask ~15% of gene expression values

CD19  GAPDH  TP53  IL7R  MS4A1  GNLY  CD14  LYZ  NKG7  FCER1A  ACTB  CD4  CD8A

[illegible] · 3 of 16 genes hidden here

expressed  [MASK] / predicted  attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15% of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values—the BERT recipe applied to single-cell biology. Click to pause.

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#18
Ask Gemini
akram baharlouei
PAUSE  REPLAY  REC

### The latent bottleneck loses information

The slide argues that compressing a cell into a latent vector can discard rare signals, affecting reconstruction and downstream tasks. It proposes preserving full expression as the motivation for flow matching.

[12:06](https://www.youtube.com/watch?v=-561cZmir5Q&t=726s) [A pipeline compresses a cell into a latent vector before reconstruction, with consequences for downstream tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7d1f23910faad49854dca7a54de1c4f6e48d5077ca51d1035b0797d09d7f363.webp)

04 · MODELING CELLS

The deeper problem: the latent bottleneck loses information

Real cell
~20,000 genes
→ Encoder → Latent vector
compressed · lossy
→ Decoder → Reconstruction
≠ the original

Downstream on the embedding
Tasks run on the latent — inheriting whatever it failed to keep.

Reconstruction is wrong
The rare, sparse signal that matters most is the first thing compressed away.

So: preserve the information
Build models that keep the full expression — which leads us to flow matching.

FROM TOKENS TO CELLS
ALTOS LABS

Interface text:
localhost:8765/From Tokens
localhost:8765/From%20Tokens%20to%20Cells%20v4.dc.html#19
Ask Gemini
akram baharlouei

## Full transcript

[0:12](https://www.youtube.com/watch?v=-561cZmir5Q&t=12s) Okay, let's get started. Uh, my name is Akram. I'm a machine learning engineer at Altos Labs. Altos Labs is a biotech startup, and the goal is to restore cell health and resilience through cellular rejuvenation to reverse disease and disabilities that can happen throughout the life.

[0:32](https://www.youtube.com/watch?v=-561cZmir5Q&t=32s) And the title of my talk is, uh, From Tokens to Cells, and this is, uh, my kind of view as someone, mm, without bio background to kind of looking into the engineering challenges of foundation models for single-cell biology.

[0:51](https://www.youtube.com/watch?v=-561cZmir5Q&t=51s) And what I want to talk about, first, what is single cell? Why do we care about single cell? How do, how do we measure it? What are the problems with the data, getting the data?

[1:01](https://www.youtube.com/watch?v=-561cZmir5Q&t=61s) And then looking at current state-of-the-art foundation models, and then some takeaways in the end.

[1:09](https://www.youtube.com/watch?v=-561cZmir5Q&t=69s) Uh, so what is single cell, and why do we care about it? I would like to start with my favorite example, Yamanaka factor. In two thousand and six, Shinya Yamanaka discovered four transcription factors that are for a specific type of, uh, proteins that when they were overexpressed in a cell, uh, they did it for, uh, in a

[1:31](https://www.youtube.com/watch?v=-561cZmir5Q&t=91s) cell, uh, which was a age-old skin cell. It could reprogram the skin cell or the skin cell to embryonic stem cell-like state. So basically from cell type skin cell old, it could reprogram it back to a young embryonic stem cell type,

[1:53](https://www.youtube.com/watch?v=-561cZmir5Q&t=113s) uh, type. And this was a breakthrough for biology, and then got him Nobel Prize later in two thousand twelve, uh, because of the application and the new chapters, possibilities for medicine, from regenerative medicine for, like, to be able to kind of regenerate tissues or organs, uh, when we can, like, reprogram a specific cell to any cell that

[2:18](https://www.youtube.com/watch?v=-561cZmir5Q&t=138s) we want to partial reprogramming and aging application. For partial reprogramming specifically is that we can also turn out that we can also only change the age of the cell.

[2:29](https://www.youtube.com/watch?v=-561cZmir5Q&t=149s) We don't have to change the type. So by changing the age of the cell, the goal is, the hope is we can restore some of youthful function that we lose throughout life as we age, and then kind of like having this as a medicine, maybe like an mRNA medicine, you know, for curing a disease that we, we,

[2:49](https://www.youtube.com/watch?v=-561cZmir5Q&t=169s) we encounter as we age. And that's, uh, one example. And then I think this year, two thousand twenty-six, we got the first type of reprogramming medicine. I think it's OSK, and, uh, it's going to be tested in a human.

[3:07](https://www.youtube.com/watch?v=-561cZmir5Q&t=187s) So it took kind of like twenty, twenty years.

[3:11](https://www.youtube.com/watch?v=-561cZmir5Q&t=191s) And, uh, and the reason, the other reason is that why we want to study, uh, a single cell is that we would like to model, to have this unified holistic view of single cell, to be able to model cell, and from there, hopefully, we can model tissue and organ and then the entire human.

[3:31](https://www.youtube.com/watch?v=-561cZmir5Q&t=211s) And there are, like, projects like Human Cell Atlas that actually started this effort, this initiative, and then they, like, you know, they have, they have mapped every single cell inside human.

[3:44](https://www.youtube.com/watch?v=-561cZmir5Q&t=224s) And the goal ultimately is something like we maybe for, uh, ultimately some, after some long time we can actually, like, actually model, uh, like, mo-model human body or there are terminologies like virtual cell, virtual tissue, virtual human, digital twins.

[4:02](https://www.youtube.com/watch?v=-561cZmir5Q&t=242s) They're all saying the same thing that the more we can model this, uh, living organisms,

[4:10](https://www.youtube.com/watch?v=-561cZmir5Q&t=250s) the better we are in understanding our body and then how

[4:14](https://www.youtube.com/watch?v=-561cZmir5Q&t=254s) we can treat a medicine, we can develop drugs.

[4:19](https://www.youtube.com/watch?v=-561cZmir5Q&t=259s) And, uh, the, the other, uh, uh, um, the other problem is that, so we, you know, I think for this audience, we know about Moore's Law. The compute is getting, like, double every year.

[4:32](https://www.youtube.com/watch?v=-561cZmir5Q&t=272s) And then we have exact opposite on drug development. Basically, the number of drugs that develop each year is kind of declining, which is surprising with all the advances in technology, in AI.

[4:45](https://www.youtube.com/watch?v=-561cZmir5Q&t=285s) And this is surprising to see. And then drug development is a field that failure is, you know, very normal. Maybe the acceptance ra-rate is kind of like, you know, five percent or even less.

[4:57](https://www.youtube.com/watch?v=-561cZmir5Q&t=297s) And, uh, when we're looking at drug development pipeline from the early, like, research and development all the way to preclinical, clinical trial and then the final, uh, stage, the whole, uh, pipeline, it could take up to ten years easily, and then it costs, it can cost billions.

[5:16](https://www.youtube.com/watch?v=-561cZmir5Q&t=316s) And the goal here is that with the advances of AI and also, like, with this, uh, virtual cell, virtual organ, virtual human, the goal is that we can kind of like reduce this time, improve, uh, reduce this timeline.

[5:30](https://www.youtube.com/watch?v=-561cZmir5Q&t=330s) And, uh, and also we're looking at the entire timeline. Let's say if we only look at like research and development, the models like, you know, protein design and stuff, we might like save like, you know, few years here, but at the end maybe it's not gonna help, uh, for the entire pipeline.

[5:47](https://www.youtube.com/watch?v=-561cZmir5Q&t=347s) So it's important to have innovation and a breakthrough, uh, across all pipeline.

[5:56](https://www.youtube.com/watch?v=-561cZmir5Q&t=356s) Now that we know single cell is im-important, how can we measure single cell? So look-looking at single, yeah, it's amazing. This is just one single cell. There is a lot going on inside that one little tiny organism.

[6:12](https://www.youtube.com/watch?v=-561cZmir5Q&t=372s) Uh, there are different modalities, and each modality is measure different things. Uh, from, um, genome, which is like, you know, DNA sequence and is almost, you know, same for all cells, to RNA sequence.

[6:25](https://www.youtube.com/watch?v=-561cZmir5Q&t=385s) RNA sequence or transcriptomic is, like, the profile of the genes that they are expressed, uh, at each time and then it, it's like a matrix. And then basically cell- it says for each cell, uh, what are the genes, what are the genes, how many genes are expressed, and then in which quantity.

[6:44](https://www.youtube.com/watch?v=-561cZmir5Q&t=404s) And is like 20,000. It's, it's 20,000 scale.

[6:51](https://www.youtube.com/watch?v=-561cZmir5Q&t=411s) And, uh, the other, uh, maybe modality is proteomics. Proteomics maybe is, like, what maybe we care about more, like, because the proteins are the one that they're doing most of the functions in our body.

[7:03](https://www.youtube.com/watch?v=-561cZmir5Q&t=423s) But the problem is that proteomics is, uh, very hard to measure and, you know, there are ways to measure it, but it's very low, uh, throughput. And morphology is about, like, the, uh, cell shape and structure, and there are, like, really good imaging technologies to kind of also measure, uh, to have, like, microscopic imaging from single cell.

[7:24](https://www.youtube.com/watch?v=-561cZmir5Q&t=444s) And, um, it's very a- also useful for a spatial, a spatial, uh, dimension, kind of like, you know, knowing the, uh, place of, like, you know, how, where a single cell is located within tissue.

[7:39](https://www.youtube.com/watch?v=-561cZmir5Q&t=459s) Um, so from all these modalities, maybe the one that has been used mostly for foundation model training, I would say, is RNA-seq. And the reason is that that the technology is, uh, eas- it's easier to measure, and the technology is like b- because of, like, PC, uh, PCR technologies and stuff, it's easier to scale.

[7:59](https://www.youtube.com/watch?v=-561cZmir5Q&t=479s) So we have, like, usually dataset for single-cell RNA in the scale of, like, tens of mil- tens of, uh, millions of cells to even, like, you know, I've heard 500 million cells and then kind of, like, even, like, there is a project, one billion cells.

[8:13](https://www.youtube.com/watch?v=-561cZmir5Q&t=493s) But usually what they talk about is, uh, RNA-seq. And then also, this is also a problem that if you really wanna understand biology and then starting from single cell, we really need to kind of have technological advances in other dimensions as well.

[8:30](https://www.youtube.com/watch?v=-561cZmir5Q&t=510s) And now having said that, let's say we got, like, you know, gathered 1 billion RNA-seq, 1 billion cells, 1 billion samples of RNA-seq ta- data. Is this good for foundation models that we can, you know, train on?

[8:43](https://www.youtube.com/watch?v=-561cZmir5Q&t=523s) And then the t- the problem is, uh, um, it's, it's very hard. The nature of the data is if you measure two identical cell at the, uh, they don't read the same.

[8:54](https://www.youtube.com/watch?v=-561cZmir5Q&t=534s) And then the problem is, uh, the problem is this is very, uh, very heterogeneous. Uh, I'll give you, give an example here that usually cells, they are living organism.

[9:05](https://www.youtube.com/watch?v=-561cZmir5Q&t=545s) They go through a lot of changes and cycles, like, from, like, you know, growing to, uh, kind of copying RNA to cell division. And, uh, usually some of these changes happens, like, in a burst, not like ...

[9:21](https://www.youtube.com/watch?v=-561cZmir5Q&t=561s) It's not something, you know, continuous. And what we are measuring with current, uh, single-cell sequencing is like a snapshot. We're taking a snapshots, uh, from, from a whole movie.

[9:36](https://www.youtube.com/watch?v=-561cZmir5Q&t=576s) Uh, now with that, and then also there is the biological reason that the data is very noisy, it's very heterogeneous. Also, there is, like, technical reasons that usually if you measure in different labs, different machines also, the data would be- the data wouldn't be the same.

[9:54](https://www.youtube.com/watch?v=-561cZmir5Q&t=594s) Now let's see how, what are ... look at some of the foundation models that we have and then see what we- they're doing with this, uh, data. Mm. And then also, like, uh, maybe a small note here that given the nature of data that is very high-dimensional, it's, like, multi-state.

[10:10](https://www.youtube.com/watch?v=-561cZmir5Q&t=610s) I think maybe, you know, some might argue that, uh, uh, quantum computing would be the, like, kind of like natural fit for this application. And then, yeah, we don't know, it might be true.

[10:21](https://www.youtube.com/watch?v=-561cZmir5Q&t=621s) Uh, but, you know, until we have quantum compute, for now we wanna see what we can get, uh, with current state-of-the-art AI.

[10:30](https://www.youtube.com/watch?v=-561cZmir5Q&t=630s) And now, uh, I wanna start with, like, transformer-based models, and there are been- they've been really good, like, you know, papers and models out there from the community. So starting from, like, scGPT and Geneformer.

[10:43](https://www.youtube.com/watch?v=-561cZmir5Q&t=643s) Uh, and what they want, they kind of like treat each, uh, cell like a sentence and, um, genes like tokens. So basically se- cell are built from genes. And what they do is, like, uh, uh, the, mm, like, you know, BERT st- a BERT-style model, like, from language that kind of, uh, masking

[11:09](https://www.youtube.com/watch?v=-561cZmir5Q&t=669s) some of the genes and the trying ... having the model to predict that, uh, them, pr- predict that genes, uh, that gene counts. So looking at the data is like a matrix, uh, and then for each cell, we have like, you know, we have, uh, the count of, uh, each genes.

[11:28](https://www.youtube.com/watch?v=-561cZmir5Q&t=688s) How many genes are, you know, how many genes are active, uh, per cell. And then it's trying to mask this to, like, um, attention, bidirectional attention across, and then kind of understand the relationship between, you know, between these genes.

[11:45](https://www.youtube.com/watch?v=-561cZmir5Q&t=705s) And, uh, this is how it's trained. And then, like, there are some, uh, uh, some, some, some downstream tasks that this model is used to. Uh, one is for, uh, so, like, you know, predicting cell type, kind of predicting, um, also, mm, perturbation response modeling.

[12:06](https://www.youtube.com/watch?v=-561cZmir5Q&t=726s) Uh, but overall, when we look into these models, what they try to do, they try to kind of get the single-cell data and then compress it in a kind of a latent vector and then do either decoding for, like, generative task or, like, classification for, uh, classif- for other task.

[12:26](https://www.youtube.com/watch?v=-561cZmir5Q&t=746s) And then what happens is when you, uh, compress this data, we're losing a lot of information. It doesn't preserve the, uh, it doesn't preserve the information. And then that's why when we're looking into this model, like, um, uh, these models, we see that sometimes, n- sometimes, like, uh, maybe, uh, s- uh, simple linear models are on par

[12:49](https://www.youtube.com/watch?v=-561cZmir5Q&t=769s) or, like, sometimes even outperforming this, uh, these, uh, models that, you know, it's like, you know, complex models with a lot of compute. It's been used to train those.

[13:00](https://www.youtube.com/watch?v=-561cZmir5Q&t=780s) And then also we had, like, two papers last year at NeurIPS. We did, like, co- comprehensive benchmarking on, uh, on these models. One is, uh, multimodal data for, like, imaging and RNA-seq, and the other one is for perturbation response modeling.

[13:17](https://www.youtube.com/watch?v=-561cZmir5Q&t=797s) So I have the links at the end of my talk. Uh, go ahead and check them out. But, like, the final, the results, uh, the, uh, what we found out was the same, uh, that also is known in the community that even though these models are very expensive to train, uh, at the end they're not performing, uh,

[13:37](https://www.youtube.com/watch?v=-561cZmir5Q&t=817s) as well comparing to, you know, other domain language, uh, and, and imaging.

[13:44](https://www.youtube.com/watch?v=-561cZmir5Q&t=824s) And this is our newer, um ... And then so transformer-based also we have, like, flow matching based models, uh, that, uh, you, you start to try to, uh, predict the distribution of the data and, uh, starting from, like, you know, Gaussian noise and then trying to, to match their distribution.

[14:03](https://www.youtube.com/watch?v=-561cZmir5Q&t=843s) And we have, uh, this paper, uh, we have this model Prime Flow, uh, that is available on arXiv, and it seems that these models, they seem to do better.

[14:16](https://www.youtube.com/watch?v=-561cZmir5Q&t=856s) There is a better comparing to trans- autoregressive based models, um, because it's tries to kind of match the distribution, predict the distribution rather than kind of, you know, understanding the da- uh, understanding the data and compressing it into, into a, uh, into, into a, a latent vector.

[14:38](https://www.youtube.com/watch?v=-561cZmir5Q&t=878s) Mm, and this is like, uh, some of the results that, uh, we have. So on the left, we see like Prime Flow, it's the greens, the green dots are, um, uh, ground truth label, and then we see that Prime Flow is kind of trying to match the distribution, whereas other models like CPAD are autoencoder-based.

[14:59](https://www.youtube.com/watch?v=-561cZmir5Q&t=899s) It's kind of like just mapping to the-- It's trying to predict the mean rather than understanding, understanding the distribution. And then we have like MMD score at the bottom.

[15:11](https://www.youtube.com/watch?v=-561cZmir5Q&t=911s) Uh, so yeah, that was, uh, uh, my talk. Three, uh, things to take away from this talk is that single cell is important, uh, and it's important to understand this.

[15:22](https://www.youtube.com/watch?v=-561cZmir5Q&t=922s) Uh, it has application for cellular rejuvenation. It's a path to like digital human, and they're helping with drug development cycle. Uh, then also we looked at, uh, uh, other-- the different modalities to measure from single cell.

[15:38](https://www.youtube.com/watch?v=-561cZmir5Q&t=938s) RNA sequence data is the one that maybe it's we have it more available in scale, and also it's measuring something important, gene expression profile. But at the same, uh, at the same time that we would like to kind of get more data, work on the scale, but it's important that to work on quality of the data of

[15:57](https://www.youtube.com/watch?v=-561cZmir5Q&t=957s) the way that we are measuring this data to be a bit more realistic of the real, uh, organism than, uh, than, than just like, you know, scale the data the way it is.

[16:09](https://www.youtube.com/watch?v=-561cZmir5Q&t=969s) And then my final, uh, conclusion is that for now it seems flow matching models, they're doing better, uh, for single-cell data, trying to match, uh, to the distribution, uh, and, uh, also to be able to, uh, scale these models, uh, that, you know, they can also do well on the data they haven't seen, they haven't trained on.

[16:31](https://www.youtube.com/watch?v=-561cZmir5Q&t=991s) We would need, you know, massive scaling, scaling of the data, I would say, and then also the quality and the way that we measure data.

[16:39](https://www.youtube.com/watch?v=-561cZmir5Q&t=999s) And then with that, yeah, we have like three papers, scGeneScope, PerturBench, last year in NeurIPS and then Prime Flow. Uh, check them out and, yeah, let me know if you have any questions.

[16:48](https://www.youtube.com/watch?v=-561cZmir5Q&t=1008s) Thank you.
