Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Dimensionality Reduction

🧠 Key takeaways

PCA is a linear dimensionality reduction technique that creates uncorrelated principal components ranked by variance, making it interpretable and efficient but less suitable for visualizing highly non-linear scRNA-seq data.

UMAP is a non-linear method that constructs and optimizes graph representations to preserve both local and global data structures, making it highly effective for visualization and clustering of single-cell data.

⚙️ Environment setup
Steps
yml

  1. Install conda:

    • Before creating the environment, ensure that conda is installed on your system.

  2. Save the yml content:

    • Copy the content from the yml tab into a file named environment.yml.

  3. Create the environment:

    • Open a terminal or command prompt.

    • Run the following command:

      conda env create -f environment.yml

  4. Activate the environment:

    • After the environment is created, activate it using:

      conda activate <environment_name>

    • Replace <environment_name> with the name specified in the environment.yml file. In the yml file it will look like this:

      name: <environment_name>

  5. Verify the installation:

    • Check that the environment was created successfully by running:

      conda env list
🗄️ Get data and notebooks

This book uses lamindb to store, share, and load datasets and notebooks using the theislab/sc-best-practices instance. We acknowledge free hosting from Lamin Labs.

  1. Install lamindb

    • Install the lamindb Python package:

    pip install lamindb

  2. Optionally create a lamin account

  3. Verify your setup

    • Run the lamin connect command:

    import lamindb as ln

ln.Artifact.connect("theislab/sc-best-practices").df()

    You should now see up to 100 of the stored datasets.

  4. Accessing datasets (Artifacts)

    • Search for the datasets on the Artifacts page

    • Load an Artifact and the corresponding object:

    import lamindb as ln
af = ln.Artifact.connect("theislab/sc-best-practices").get(key="key_of_dataset", is_latest=True)
obj = af.load()

    The object is now accessible in memory and is ready for analysis. Adapt the lamindb.Artifact.connect("theislab/sc-best-practices").get("SOMEIDXXXX") suffix to get respective versions.

  5. Accessing notebooks (Transforms)

    lamin load <notebook url>

    which will download the notebook to the current working directory. Analogously to Artifacts, you can adapt the suffix ID to get older versions.

scRNA-seq is a high-throughput sequencing technology that produces datasets with high dimensions in the number of cells and genes. Therefore, scRNA-seq data suffers from the ‘curse of dimensionality’.

Curse of dimensionality

The Curse of dimensionality was first brought up by R. Bellman Bellman et al., 1957 and describes that, in theory, high-dimensional data contains more information, but in practice this is not the case. Higher-dimensional data often contains more noise and redundancy, and therefore, adding more information does not provide benefits for downstream analysis steps.

Not all genes are informative and therefore not all genes are essential for tasks such as clustering. We already aimed to reduce the dimensionality of the data with feature selection. As a next step, we will further reduce the dimensions of single-cell RNA-seq data with dimensionality reduction algorithms. These algorithms are an important step during preprocessing to reduce the data complexity and for visualization.

Dimensionality reduction

Figure 1:Dimensionality reduction embeds the high-dimensional data into a lower-dimensional space. The low-dimensional representation still captures the underlying structure of the data while having as few dimensions as possible. Here we visualize a three-dimensional object projected into two dimensions.

Xing et al. compared in an independent comparison the stability, accuracy, and computing cost of 10 different dimensionality reduction methods Xiang et al., 2021. They propose to use t-distributed stochastic neighbor embedding (t-SNE) as it yielded the best overall performance. Uniform manifold approximation and projection (UMAP) showed the highest stability and separated the original cell populations best. An additional dimensionality reduction worth mentioning in this context is principal component analysis (PCA), which is still widely used.

Generally, t-SNE and UMAP are very robust and mostly equivalent if specific choices for the initialization are selected Kobak & Berens, 2019.

All aforementioned methods are implemented in scanpy.

Now we start with importing all required Python packages and load the dataset after quality control, normalization, and feature selection.

Running this on a GPU

PCA, t-SNE, UMAP, and the neighbor graph are among the steps that benefit most from GPU acceleration. On large datasets, rapids-singlecell drops their runtime by 1–2 orders of magnitude with a near-identical API.

import lamindb as ln
import scanpy as sc

# Suppress verbose logging from Scanpy
sc.settings.verbosity = 0

# Set figure parameters for clean, minimal plots
sc.settings.set_figure_params(dpi=80, facecolor="white", frameon=False)

assert ln.setup.settings.instance.slug == "theislab/sc-best-practices"

ln.track()
Output
→ connected lamindb: theislab/sc-best-practices
→ loaded Transform('liGMVGre4G5H0000'), re-started Run('ZFh04bqi...') at 2025-05-13 03:27:34 UTC
→ notebook imports: lamindb==1.3.2 scanpy==1.11.1

af = ln.Artifact.connect("theislab/sc-best-practices").get(
    key="preprocessing_visualization/s4d8_feature_selection.h5ad", is_latest=True
)
adata = af.load()

We will use a normalized representation of the dataset for dimensionality reduction and visualization, specifically the shifted logarithm.

Why should you use normalized data for dimensionality reduction?

First, gene expression data is typically heavily right-skewed with many zeros and a few highly expressed genes. Log transformation helps normalize this distribution, making it more symmetric and reducing the impact of extreme values. This prevents highly expressed genes from dominating any distance calculations.

Second, biological signals in expression data often follow multiplicative rather than additive patterns. Log transformation converts multiplicative relationships to additive ones, which most dimensionality reduction algorithms (PCA, t-SNE, UMAP) are designed to detect.

Third, variance in raw expression data typically scales with mean expression levels. Log transformation stabilizes this variance, preventing highly expressed genes from dominating the analysis. For zero values (dropout events common in single-cell data), a pseudocount is typically added before log transformation (log1p) to avoid mathematical errors.

adata.X = adata.layers["scran_normalization"]

We start with:

PCA

In our dataset, each cell is a vector of a n_var-dimensional vector space spanned by some orthonormal basis. As scRNA-seq suffers from the ‘curse of dimensionality’, we know that not all features are important to understand the underlying dynamics of the dataset and that there is an inherent redundancyGrün et al., 2014. PCA creates a new set of uncorrelated variables, so-called principal components (PCs), via an orthogonal transformation of the original dataset. The PCs are linear combinations of features in the original dataset and are ranked in decreasing order of variance to define the transformation. In the ranking, the first PC typically captures the greatest amount of variance. PCs with the lowest variance are discarded to effectively reduce the dimensionality of the data without losing information.

PCA offers the advantage that it is highly interpretable and computationally efficient. However, as scRNA-seq datasets are rather sparse due to dropout events and therefore highly non-linear, visualization with the linear dimensionality reduction technique PCA is not very appropriate. PCA is typically used to select the top 10-50 PCs, which are used for downstream analysis tasks.

# setting highly variable as highly deviant to use scanpy 'use_highly_variable' argument in sc.pp.pca
adata.var["highly_variable"] = adata.var["highly_deviant"]
sc.pp.pca(adata, svd_solver="arpack", mask_var="highly_variable")
sc.pl.pca_scatter(adata, color="total_counts")
<Figure size 320x320 with 2 Axes>
How to understand a PCA scatter plot

PCA looks for the directions in the data where the values of the cells (or samples) differ the most.

  1. Axes: PC1 and PC2 are the first and second principal components(directions). PC1 is the direction where the data is most spread out, and PC2 is the next most variable direction, perpendicular to PC1.

  2. Points: Each point is a single cell. The color of each point represents the total number of transcripts (UMIs) detected in that cell.

By projecting the data onto PC1 and PC2, we capture most of the meaningful variation in fewer dimensions.

Although we do not aim to interpret biological meaning directly from this PCA plot at this stage, we can observe cellular heterogeneity: cells with similar expression profiles tend to cluster together, which may reflect different cell types or states.

t-SNE

t-SNE is a graph based, non-linear dimensionality reduction technique which projects the high dimensional data onto 2D or 3D components. The method defines a Gaussian probability distribution based on the high-dimensional Euclidean distances between data points. Subsequently, a Student t-distribution is used to recreate the probability distribution in a low dimensional space where the embeddings are optimized using gradient descent.

sc.tl.tsne(adata, use_rep="X_pca")
sc.pl.tsne(adata, color="total_counts")
<Figure size 320x320 with 2 Axes>
How to understand t-SNE plot

  1. Axes have no meaning!

  2. Points: Each point represents a single cell. Color represents the total number of transcripts (UMIs) detected in that cell.

This plot helps reveal cellular subpopulations: tight clusters may correspond to distinct cell types or states. Unlike PCA, which preserves global variance, t-SNE emphasizes local structure. For this reason, it is best practice to initialize t-SNE from PCA to reduce noise and improve stability of the embedding.

UMAP

UMAP is a graph-based, non-linear dimensionality reduction technique and is principally similar to t-SNE. It constructs a high-dimensional graph representation of the dataset and optimizes the low-dimensional graph representation to be as structurally similar as possible to the original graph.

We first calculate PCA and subsequently create a neighborhood graph based on our data.

sc.pp.neighbors(adata)
sc.tl.umap(adata)
sc.pl.umap(adata, color="total_counts")
<Figure size 320x320 with 2 Axes>
How to understand a UMAP plot

  1. Axes have no meaning!

  2. Points: Each point represents a single cell.
    Color indicates the total number of transcripts (UMIs) detected in that cell.

UMAP emphasizes both local and some global structure, often better preserving the overall shape of the data compared to t-SNE, i.e. similar clusters are closer together. It constructs a neighborhood graph based on PCA-reduced data, then optimizes a low-dimensional layout that reflects relationships in the original high-dimensional space. UMAP is widely used for visualizing cell clusters and trajectories in single-cell data.

Inspecting quality control metrics

We can now also inspect the quality control metrics we calculated previously in our PCA, TSNE or UMAP plot and potentially identify low-quality cells.

sc.pl.umap(
    adata,
    color=["total_counts", "pct_counts_mt", "scDblFinder_score", "scDblFinder_class"],
)
<Figure size 1545.6x320 with 7 Axes>

Cells with a high doublet score are projected to the same region in the UMAP. We will keep them in the dataset for now but generally this suggests to potentially re-visit the quality control strategy again to ensure that as few doublets as possible are retained.

af = ln.Artifact(
    adata,
    key="preprocessing_visualization/s4d8_dimensionality_reduction.h5ad",
    description="anndata after dimensionality reduction",
).save()
af
Output
→ creating new artifact version for key='preprocessing_visualization/s4d8_dimensionality_reduction.h5ad' (storage: 's3://lamin-eu-central-1/VPwcjx3CDAa2')
! The cache path /Users/seohyon/Library/Caches/lamindb/lamin-eu-central-1/VPwcjx3CDAa2/preprocessing_visualization/s4d8_dimensionality_reduction.h5ad already exists, replacing it.
Artifact(uid='6MXF3Uy8DkG9wy720002', is_latest=True, key='preprocessing_visualization/s4d8_dimensionality_reduction.h5ad', description='anndata after dimensionality reduction', suffix='.h5ad', otype='AnnData', size=4530494307, hash='E-h2bZDOxSnEFWk1reXptw', space_id=1, storage_id=1, run_id=10, created_by_id=5, created_at=2025-05-13 03:31:03 UTC)

Contributors

We gratefully acknowledge the contributions of:

Authors

  • Anna Schaar

  • Seo H. Kim

Reviewers

  • Lukas Heumos

References
  1. Bellman, R., Bellman, R. E., & Corporation, R. (1957). Dynamic Programming. Princeton University Press. https://books.google.de/books?id=rZW4ugAACAAJ
  2. Xiang, R., Wang, W., Yang, L., Wang, S., Xu, C., & Chen, X. (2021). A Comparison for Dimensionality Reduction Methods of Single-Cell term`RNA`-seq Data. Frontiers in Genetics, 12. 10.3389/fgene.2021.646936
  3. Kobak, D., & Berens, P. (2019). The art of using t-SNE for single-cell transcriptomics. Nature Communications, 10(1), 5416. 10.1038/s41467-019-13056-x
  4. Grün, D., Kester, L., & Van Oudenaarden, A. (2014). Validation of noise models for single-cell transcriptomics. Nature Methods, 11(6), 637–640.
Preprocessing and visualization
Feature selection
Identifying cellular structure
Clustering