k-means clustering

Most of the world's data has no label. k-means is the clustering method you should be able to derive on a whiteboard: pick k centers, assign each point to the nearest, recompute the centers, repeat.

The big idea

1. Initialize k centroids (preferably with k-means++, which spreads them out).
2. Assign each point to the nearest centroid (usually Euclidean distance).
3. Recompute each centroid as the mean of assigned points.
4. Repeat until assignments stabilize.

You must pick k. Sensitive to outliers (they pull centroids). Fails on non-spherical shapes (rings, moons) under Euclidean distance. Random init can matter, so use k-means++ and multiple n_init.

Walking through choosing k

• Elbow: plot inertia (within-cluster SS) versus k. Look for a diminishing-returns bend.
• Silhouette score: mean of (b − a) / max(a, b) where a is mean intra-cluster distance and b is mean distance to the nearest other cluster. Higher is better. Negative means mis-assignment.
• Downstream usefulness: the real test. Do clusters map to actions (sales plays, topic queues)?

300 points from three Gaussian blobs. At k=3, inertia drops sharply from k=1→3 then flattens. Silhouette peaks near k=3. At k=10 inertia is lower still but clusters fragment a single blob — worse for product segmentation.

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

X, _ = make_blobs(n_samples=600, centers=4, cluster_std=0.8, random_state=0)
X = StandardScaler().fit_transform(X)

for k in range(2, 6):
    km = KMeans(n_clusters=k, n_init=10, random_state=0)
    labels = km.fit_predict(X)
    print(k, "inertia", round(km.inertia_), "sil", round(silhouette_score(X, labels), 3))

Typical output: silhouette peaks at k=4 (the true number of blobs), while inertia keeps falling.

Gaussian mixture models (GMMs) soft-assign points to k Gaussians with different covariances. A point can be 70% cluster A and 30% cluster B, and clusters can be elliptical. Fit with expectation–maximization. Useful when you believe overlapping populations. More parameters than k-means, so it needs more data. For very large n, mini-batch k-means usually wins on pragmatism.

Use k-means when you want every point assigned, roughly blob-shaped groups, and a fixed k for capacity planning (for example 8 sales segments). Use GMM when soft membership or elliptical shapes matter.

Modern pipelines often cluster embeddings, not raw columns: embed tickets, L2-normalize, run k-means, label clusters by reading exemplars. Cosine similarity is the usual friend of normalized embeddings. Euclidean k-means on L2-normalized vectors is closely related. PCA to 50–100 dims before k-means can denoise when embeddings are 768–1536-d.

When clusters are used as features for a supervised model, fit on train only, then predict on val/test — same leakage discipline as any other transform.

Common mistakes

• Skipping scaling. One large-range column owns the distance.
• Picking k only from silhouette when sales can only run four plays. Merge to the number you can act on.
• One random init. Use k-means++ and several n_init.

Check yourself

1. Why scale features before k-means? Answer: Euclidean distance is dominated by large-range columns unless you standardize.
2. Inertia is lower at k=10 than k=3 on three blobs. Should you pick 10? Answer: No. Extra clusters fragment a real group. Use the elbow, silhouette and the action you can take.

Key takeaways

• k-means assigns points to k centroids and updates those centroids as means.
• Choose k with elbow, silhouette and downstream utility. Scale inputs.
• k-means++ and multiple inits reduce unlucky starts.
• Embeddings plus k-means is the usual topic-discovery pipeline today.

Further learning

Next: DBSCAN and hierarchical clustering.

k-means geometry: