Math toolkit
Probability, statistics and information
The probability ideas behind ML losses, Bayes updates and honest uncertainty. Start here for the map of the section.
A model that outputs 0.8 for "fraud" is making a probability claim. Cross-entropy loss, softmax, temperature sampling and confidence intervals all come from a small set of ideas: random variables, likelihood, Bayes and information. Learn them and the losses in training code stop looking like arbitrary formulas.
Training works like this. You define a probabilistic model of the labels, turn its likelihood into a loss, differentiate that loss with calculus, and descend with optimization. The concepts below cover the probability half of that story.
Every number a model prints is a guess with uncertainty attached. These ideas tell you where the guess comes from and how much weight it deserves. They also answer practical questions. Is a 1-point gain real? Does a positive test mean the condition is likely? Why did the team choose this loss?
Concepts in this section
Read them in order. The first two give you the vocabulary. The rest use it.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Random variables and distributions | What is a probability distribution, and what do expectation and variance measure? |
| 2 | Conditional probability and Bayes | How should a test result change your belief? |
| 3 | Maximum likelihood | Why do squared error and cross-entropy show up as losses? |
| 4 | Softmax and temperature | How do raw scores become probabilities, and what does temperature change? |
| 5 | Entropy and cross-entropy | How do you measure surprise, and what is perplexity? |
| 6 | Uncertainty and sample size | How sure can you be about a metric from a finite test set? |
Common mistakes
- Reading accuracy as the chance a positive result is right. The base rate changes what a positive means. A 99%-accurate test on a 1% condition gives a positive result that is right only about 17% of the time.
- Reporting a metric with no sample size. A small eval set can swing by more than the gain you are celebrating.
- Treating a loss as arbitrary. Most standard losses are maximum likelihood under a specific assumption about the data.
- Multiplying small probabilities directly. Long products underflow to zero. Work in logs, where products become sums.
- Swapping the two directions of a conditional.
P(positive test | sick)andP(sick | positive test)are different numbers.
Key takeaways
- Most supervised outputs are a Bernoulli, categorical or Gaussian distribution.
- Bayes' rule updates a prior with evidence. Base rates matter as much as accuracy.
- Squared error is maximum likelihood under Gaussian noise. Cross-entropy is maximum likelihood for categorical outputs.
- Softmax turns logits into probabilities, and temperature sharpens or flattens them.
- Report eval metrics with an interval or the sample size.
Further learning
Seeing Theory (Brown University) has interactive visuals for distributions, conditional probability and Bayes. StatQuest with Josh Starmer is a good companion. Mathematics for Machine Learning covers the math formally. Think Stats (Allen Downey) is short and code-first.
Related
Random variables and distributions
A random variable is a value you cannot know until you observe it. A distribution says how likely each value is.
Maximum likelihood
Maximum likelihood picks the parameters that make the data you saw most probable. Squared error and cross-entropy are both this one idea.
Linear and logistic regression
The two models under almost everything in ML. They fit lines, predict probabilities and stay honest through regularization.
Evaluating models
Choose metrics that match the decision, set thresholds from costs, check calibration, and guard against leakage. A high offline score means little until it holds up on real decisions.
Sampling and decoding
Temperature, top-p, top-k and other knobs that turn logits into text, and when to touch them.