PCA for machine learning

Principal component analysis (PCA) finds orthogonal directions of maximum variance. Projecting onto the first r components gives a low-dimensional representation that keeps as much variance as possible. It is lossy compression and, often, denoising.

The big idea

If X is centered (n × d), the top principal components are the top right singular vectors of X (or eigenvectors of the covariance).

explained_variance_ratio_i = σ_i² / Σ_j σ_j²

Rules of thumb, then verify:

• Keep components until cumulative explained variance hits about 90–95% for compression before a supervised model.
• Keep 2–3 for visualization even if they explain less. Eyes need 2D.
• Check a scree plot. A sharp elbow means a natural rank.

PCA is linear. Nonlinear manifolds (a Swiss roll of embeddings) need UMAP or autoencoders for structure. Still use PCA first as a baseline projection. If PCA already separates classes in 2D, your supervised problem may be easy.

Whitening divides each PC by its singular value so variances become equal. Sometimes helps downstream methods that assume isotropic noise. Often hurts when variance is the signal. Default: project without whitening unless a method's docs ask for it.

Walking through a 50-feature table

50 features, first three PCs explain 0.41 + 0.22 + 0.11 = 0.74 of variance. A logistic regression on 3 PC scores may beat a noisy 50-feature model on small n. You lose interpretability of original columns unless you inspect loadings (V).

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
X = rng.normal(size=(400, 50))
X[:, 1] += 2 * X[:, 0]  # one strong shared direction
X = StandardScaler().fit_transform(X)
pca = PCA(n_components=10).fit(X)
print(np.round(pca.explained_variance_ratio_[:3], 3))
print("cumsum3", round(pca.explained_variance_ratio_[:3].sum(), 3))

Typical output: the first component captures the duplicated direction, and the first three already hold a large share of variance.

t-SNE and UMAP build 2D/3D views that keep local neighborhoods. They are for eyes, not for clustering input or distance math downstream.

Common misreadings:

• Cluster sizes and between-cluster distances in the plot are not trustworthy.
• Re-running with different perplexity or random seeds changes the picture.
• You cannot "project a new point" through classic t-SNE without extra machinery.

Do not feed t-SNE coordinates into a production classifier. Use PCA or learned embeddings for features. Use t-SNE/UMAP only in notebooks to sanity-check structure.

PCA to 50–100 dims before k-means can denoise and speed distance computations when embeddings are 768–1536-d.

Common mistakes

• Skipping centering. PCA on uncentered data is dominated by the mean, not the variance.
• Shipping t-SNE coordinates as features. They are a picture.
• Keeping 2 components because the plot looks nice, then training on only those 2 when they explain 12% of variance. Visualization r and model r are different jobs.

Check yourself

1. First three PCs explain 0.41, 0.22 and 0.11. How much variance is kept? Answer: 0.74.
2. Why not train a classifier on t-SNE coordinates? Answer: Distances and cluster sizes in the plot are not trustworthy, and new points do not project cleanly.

Key takeaways

• PCA/SVD compresses variance into orthogonal components for visualization and features.
• Keep enough components for the job: ~90–95% for compression, 2–3 for a plot.
• t-SNE and UMAP are visual tools. Do not treat plot distances as data.
• Center (and usually scale) before PCA.

Further learning

Next: anomaly detection.

StatQuest’s PCA walkthrough:

scikit-learn: Decomposing signals in components (PCA)Principal component analysis (PCA): Exact PCA and probabilistic interpretation: PCA is used to decompose a multivariate dataset in a set of successive orthogonal components that explain a maximum a...scikit-learn