Logistic regression and log-odds

Classification needs a probability between 0 and 1, not an unbounded score. Logistic regression keeps the linear score and passes it through the sigmoid function. The output is a probability, and training uses log loss.

The big idea

Compute the linear score z = w · x + b. Then apply the sigmoid:

p = σ(z) = 1 / (1 + e^(−z))

The sigmoid squashes any score into the range (0, 1). A large positive z gives p near 1, and a large negative z gives p near 0.

The decision boundary is where p = 0.5, which is exactly where z = 0. That is a straight line or hyperplane in feature space.

Training minimizes log loss, also called binary cross-entropy:

L = −[ y · log p + (1 − y) · log(1 − p) ]

It is the negative log-likelihood of the labels, so training finds the weights that make them most probable.

Walking through it

Use one feature, with weight w = 2 and bias b = −3.

• At x = 2: z = 2·2 − 3 = 1, so p = σ(1) ≈ 0.731.
• At x = 0: z = −3, so p = σ(−3) ≈ 0.047.
• The boundary sits at x = 1.5, where z = 0.

The 0.5 threshold is a default, not a law. If a false positive is expensive, you might label a case positive only when p ≥ 0.8. The threshold is a product choice, separate from training.

Coefficients are log-odds

The odds of an event are p / (1 − p). Taking the log of the odds gives the linear score:

log(p / (1 − p)) = w · x + b

So a coefficient w_j tells you that a one-unit increase in feature j multiplies the odds by e^(w_j), holding the other features fixed.

A churn model has two features. w_logins = −0.40 means one extra login multiplies the odds of churn by e^(−0.40) ≈ 0.67. w_tickets = 0.25 means one extra support ticket multiplies the odds by e^0.25 ≈ 1.28.

Here is the odds arithmetic for a full example. The baseline churn rate is 8%, so the odds are 0.08 / 0.92 ≈ 0.087, and the log-odds are about −2.44. Add a standardized ticket feature with weight 0.60. For an account one standard deviation above average on tickets:

log-odds = −2.44 + 0.60 = −1.84
odds     = e^(−1.84) ≈ 0.159
p        = 0.159 / 1.159 ≈ 0.137

Churn risk moves from 8% to about 14%. The odds nearly doubled, because 0.159 / 0.087 ≈ 1.8.

Why not MSE for classification?

You could code labels as 0 and 1 and minimize squared error. That fails for two reasons. The output is not guaranteed to stay between 0 and 1. And log loss is a proper scoring rule for probabilities, so it rewards honest probabilities. Use MSE for continuous targets and log loss for classes.

For more than two classes, use one weight vector per class and the softmax function. It turns the scores into probabilities that add up to 1.

Common mistakes

• Reading e^w as a change in probability. It multiplies the odds. The change in probability depends on the starting probability.
• Keeping the 0.5 threshold by default. Choose the threshold from the costs of each kind of error.
• Using squared error on 0/1 labels. The outputs can leave the range 0 to 1.

Check yourself

1. A coefficient is w = 0.693. What does a one-unit increase do to the odds? Answer: It multiplies them by e^0.693 ≈ 2, so the odds roughly double.
2. At what value of z is p = 0.5? Answer: z = 0. That is where the decision boundary sits.

Key takeaways

• The sigmoid turns a linear score into a probability.
• Log loss is the training objective, and its gradient is (p − y) · x.
• Coefficients are log-odds: e^w is the odds multiplier for a one-unit change.
• The decision threshold is a product choice, not a training choice.

Further learning

Josh Starmer's walkthrough of logistic regression intuition, before the coefficient details:

How to read logistic coefficients as log-odds, the piece most engineers skip:

Next, control the size of the weights: regularization with L1, L2 and elastic net.