Guide

Math toolkit

Random variables and distributions

A random variable is a value you cannot know until you observe it. A distribution says how likely each value is.

15 of 187 · 2 min read

A model that predicts a click is making a claim about an uncertain outcome. Before you can talk about losses or uncertainty, you need a shared vocabulary for "a value we don't know yet."

Concept

A random variable is a quantity whose value is uncertain until you observe it: a die roll, whether a user clicks, the next token an LLM emits. A probability distribution assigns a probability to each possible value.

Discrete and continuous

Outcomes come in two kinds, and each has its own tool.

  • PMF (probability mass function) is for discrete outcomes. It gives P(X = k) for each k, and the values add up to 1.
  • PDF (probability density function) is for continuous outcomes. Probabilities are areas under the curve. A density at one point can be bigger than 1. Only areas count as probabilities.

Expectation and variance

Expectation is the average value over many repeats:

E[X] = sum of x · P(x)      (discrete)

Variance measures spread: Var(X) = E[(X − E[X])²], the average squared distance from the mean.

Walking through a click

Let X be 1 if a user clicks and 0 if not, with P(click) = 0.3. This is a Bernoulli distribution.

E[X]   = 0 · 0.7 + 1 · 0.3 = 0.3
E[X²]  = 0.3          (X is 0 or 1, so X² = X)
Var(X) = E[X²] − (E[X])² = 0.3 − 0.09 = 0.21

For any Bernoulli with probability p, the variance is p(1 − p). Here that is 0.3 × 0.7 = 0.21, which matches.

Distributions you will meet

NameSupportRole in ML
Bernoulli{0, 1} with P(1) = pOne yes/no label, such as a click
CategoricalK classes with probabilities p₁…pₖMulticlass labels and softmax outputs
Gaussian (Normal)The real line, mean μ, variance σ²Regression noise and many latent models
BinomialNumber of successes in n Bernoulli trialsTotal clicks out of n impressions
UniformFlat over an intervalRandom initialization and dropout masks

The Gaussian has this density:

p(x) = (1 / √(2πσ²)) · exp( −(x − μ)² / (2σ²) )

Common mistakes

  • Reading a density as a probability. A PDF value can exceed 1. Only the area under the curve is a probability.
  • Confusing the expectation with a typical value. For a Bernoulli with p = 0.3, E[X] = 0.3, yet X is never 0.3.
  • Forgetting that probabilities must add up. A PMF whose values do not sum to 1 is not a distribution.

Check yourself

  1. What is the expectation of a fair six-sided die? Answer: 3.5, because (1 + 2 + … + 6) / 6 = 21 / 6.
  2. Why is the variance of a constant 0? Answer: nothing varies, so every squared distance from the mean is 0.

Key takeaways

  • A random variable is uncertain until observed; a distribution assigns probabilities to its values.
  • Discrete outcomes use a PMF. Continuous outcomes use a PDF, where only areas are probabilities.
  • Expectation is the long-run average. Variance measures spread.
  • Bernoulli, categorical and Gaussian cover most ML output spaces.

Further learning

Next, learn how new evidence changes a probability in conditional probability and Bayes.

Conditional probability and Bayes

How learning one fact changes the probability of another, and how Bayes' rule runs that change in reverse. Base rates often matter more than test accuracy.

Maximum likelihood

Maximum likelihood picks the parameters that make the data you saw most probable. Squared error and cross-entropy are both this one idea.

Probability, statistics and information

The probability ideas behind ML losses, Bayes updates and honest uncertainty. Start here for the map of the section.