Regularization with L1, L2 and elastic net

Without a penalty, a linear model can put huge weights on noisy or redundant features and overfit. Regularization adds a term to the loss that favors small weights. The model gives up a little fit on the training data in exchange for weights that hold up on new data.

The big idea

The training objective becomes:

loss = data loss + λ · penalty

Here λ (lambda) sets how strong the penalty is. There are three common penalties:

In scikit-learn the knob is usually C = 1/λ. A smaller C means stronger regularization.

An analogy: L2 is a pull toward zero that gets weaker as a weight gets smaller. L1 is a constant pull, so it can park a weight exactly at zero. The analogy stops at correlated features, where L1 picks one feature of a group and ignores the rest.

Walking through a collinear pair

Suppose two features are copies of each other, x₁ = x₂, and the target is y = x₁. Any weights with w₁ + w₂ = 1 fit the data equally well. The data cannot say which split is right.

L2 breaks the tie by choosing the smallest total of squared weights. Minimize w₁² + w₂² subject to w₁ + w₂ = 1, and the answer is w₁ = w₂ = 0.5. The weight is shared evenly.

L1 gives every split the same penalty, so the choice is arbitrary. In practice the solver tends to put all the weight on one of the two features.

This matters in real data. Two near-duplicate columns, such as spend over the last 30 days and spend over the last 31 days, can give wildly different coefficients from one resample to the next. Regularization makes the coefficients stable enough to read.

Choosing the strength

Pick λ (or C) on the validation set. Sweep it over a log grid, and plot the validation score against it. You usually see a U-shape:

• A tiny C underfits, and most weights sit near zero.
• A huge C overfits, and noisy features get large weights.
• The validation minimum is your operating point.

Common mistakes

• Choosing λ on the test set. The test score then becomes optimistic. Tune on validation, and report test once.
• Penalizing unscaled features. The penalty depends on the feature scale, so a feature in the millions gets treated very differently from one in single digits. Scale first. See feature scaling.
• Reading a zeroed L1 weight as "useless". The feature may be correlated with another feature that the model kept.

Check yourself

1. Two identical features both fit y with w₁ + w₂ = 1. Which weights does L2 pick? Answer: w₁ = w₂ = 0.5, because that minimizes the sum of squares.
2. You decrease C. What happens to the weights? Answer: The regularization gets stronger, so the weights shrink toward zero.

Key takeaways

• Regularization adds a penalty for large weights, which reduces overfitting.
• L2 shrinks weights smoothly and shares weight across correlated features.
• L1 can set weights to exactly zero, which gives sparse models.
• Tune the strength on validation data, never on test data.

Further learning

Next, make the features comparable before you penalize them: feature scaling and leakage-safe pipelines.