Conditional probability and Bayes

A test result changes what you believe about a patient. A user's country changes how likely they are to click. Conditional probability measures that change, and Bayes' rule lets you flip the direction of the question.

Two events are independent when P(A, B) = P(A) · P(B). Knowing one tells you nothing about the other.

Bayes' rule

Bayes' rule flips a conditional:

P(H|E) = P(E|H) · P(H) / P(E)

• P(H) is the prior: your belief in hypothesis H before the evidence.
• P(E|H) is the likelihood: how probable the evidence is if H is true.
• P(H|E) is the posterior: your belief after the evidence.
• P(E) is the evidence, the normalizing sum Σ P(E|Hᵢ) P(Hᵢ).

Walking through a medical test

Disease prevalence is 1%, so P(D) = 0.01. The test catches 99% of sick people, so P(+|D) = 0.99. It flags 5% of healthy people, so P(+|¬D) = 0.05. You test positive. What is P(D|+)?

P(+)   = P(+|D)·P(D) + P(+|¬D)·P(¬D)
       = 0.99 · 0.01  + 0.05 · 0.99
       = 0.0099        + 0.0495
       = 0.0594

P(D|+) = 0.99 · 0.01 / 0.0594 ≈ 0.167, about 17%

Counting people makes this easier to believe. Take 10,000 people:

• 100 are sick, and 99 of them test positive.
• 9,900 are healthy, and 5% of them (495) test positive.
• So 594 people test positive, and only 99 of them are sick: 99 / 594 ≈ 17%.

The healthy crowd is so large that its false alarms outnumber the true cases.

Common mistakes

• Ignoring the prior. A 99%-accurate test on a 1% condition still gives mostly false alarms.
• Swapping P(A|B) and P(B|A). The chance of a positive test given disease is not the chance of disease given a positive test.
• Assuming independence. Check that two features really do not move together before you multiply their probabilities.
• Trusting a feature that is too predictive. If a feature is almost a copy of the label and is not available at serving time, the model looks brilliant offline and fails in production. See feature engineering.

Check yourself

1. With the same test, why is P(D|+) far below 99%? Answer: 495 healthy people test positive, while only 99 sick people do.
2. If the prior rose to 10%, would P(D|+) go up or down? Answer: up, to about 69%, because 0.099 / 0.144 ≈ 0.69.

Key takeaways

• Conditional probability is P(A|B) = P(A, B) / P(B).
• Bayes' rule flips a conditional using the prior and the likelihood.
• Base rates can dominate test accuracy.
• Independence is a claim to check, not a default.

Further learning

Grant Sanderson's geometric Bayes video makes the prior-versus-likelihood trade visual:

Next, see how the same probability ideas explain common losses in maximum likelihood.