Framing a business problem as a prediction

Most machine learning projects fail before the first model is trained, and the cause is usually the framing. The target is wrong, the labels do not match the decision, or a rule would have solved the problem. Framing is the work of turning a business decision into a prediction that a model can learn. Write it down as four sentences before you open a notebook.

The four sentences

1. Prediction target. The exact quantity or class the model outputs.
2. Label source. Where the ground truth comes from, and how noisy it is.
3. Decision. What a person or system does with the prediction.
4. Cost of errors. The cost of a false positive versus a false negative, in money, risk or user trust.

If you cannot write the fourth sentence, the costs are not yet understood, and the threshold cannot be chosen.

Walking through a churn example

"Use ML for churn" is too vague to build. Here is the reframe:

1. Target. The probability that an active account cancels within 30 days, scored each morning.
2. Label. Actual cancellations in the following 30 days. Features use only events before the morning of scoring.
3. Decision. Which accounts get a retention call.
4. Costs. A wasted call costs about $40. Losing an account worth $2,400 a year is 60 times that cost.

That asymmetry says to favor recall, catching most of the accounts that will leave, at a tolerable level of false alarms. Raw accuracy would tell you nothing useful here. Always predicting "stays" scores high accuracy and catches no one.

Common framing traps

• Proxy labels. You want "the user is happy", but you only have star ratings from the people who bother to rate. The model learns who rates, not who is happy.
• Actionable versus interesting. Predicting a cancellation after it happened is accurate and useless. Predict early enough to act.
• Confounded interventions. If you train on past retention offers, the labels reflect those offers. Offline results can mislead about a new policy, so test it online.

Check yourself

1. Churners are 8% of accounts. Why is 92% accuracy from "always stays" a poor result? Answer: It catches zero churners, so it fails at the decision the model is for.
2. A fraud model's false declines and false passes have different costs. Where should that difference show up? Answer: In the decision threshold, not only in the training loss.

Key takeaways

• Write the target, label source, decision and error costs before training.
• Name the decision. If there is no decision, there is no ML problem yet.
• Choose the metric and threshold from the costs of each error.
• Watch for proxy labels, late predictions and confounded past policies.

Further learning

Next, split the data in a way that matches how the model will be used: train, validation and test splits.