Linear and logistic regression

Every neural network ends in a linear layer, and every boosted-tree baseline gets compared against a logistic model. Linear regression and logistic regression are the two models you should be able to derive, implement, regularize and interpret. They do not always win on tabular data, because trees often do better. They are still the language the rest of ML is written in.

Both models share one shape: a weighted sum of features, a loss, and a way to choose the weights. Framing tells you what to predict. These pages cover how to fit the model and how to read it.

Where it shows up

A neural network's final layer is a linear model on learned features. A logistic baseline is the first model most teams should ship and measure. Regularization, scaling and the log-odds reading all carry over to larger models. Learn them here, where the math is small enough to check by hand.

Concepts

When linear models win and when they lose

Linear and logistic models win when you need calibrated probabilities, coefficients you can explain, a fast and boring baseline, or a dot product that fits a tight latency budget. They also win when a regulator or a customer asks why a decision was made.

They lose when interactions and thresholds dominate, such as "if tenure is under 7 days and tickets are over 3, churn". You can add those interaction features by hand, but gradient-boosted trees often win on tabular data without that work. Neural nets win when the raw input is pixels, audio or tokens.

Common mistakes

• Reading coefficients as causes. A coefficient describes association within the model. Check the scale and the correlation between features before you explain it.
• Choosing the penalty strength on the test set. Tune on validation, and report test once.
• Fitting a scaler before the split. The scaler then sees test statistics. Put it inside a pipeline.

Key takeaways

• Linear regression minimizes squared error. Its closed form is exact, and gradient descent scales further.
• Logistic regression maps a linear score through a sigmoid to a probability, and it trains with log loss.
• Coefficients are log-odds. e^w is the odds multiplier for a one-unit change.
• Regularize with L2 by default. Use L1 when you want sparsity, and tune on validation data.

Further learning

An Introduction to Statistical Learning covers linear and logistic regression with clear labs:

An Introduction to Statistical LearningAn Introduction to Statistical Learning

The scikit-learn user guide covers the estimators, penalties and pipelines used here:

scikit-learn User GuideSupervised learning- Linear Models- Ordinary Least Squares, Ridge regression and classification, Lasso, Multi-task Lasso, Elastic-Net, Multi-task Elastic-Net, Least Angle Regression, LARS Lasso, Or...scikit-learn

Stanford's machine learning notes derive these models from maximum likelihood:

Stanford: Machine LearningCS229: Machine Learningcs229.stanford.edu