Classical machine learning
Features and data preparation
Turning raw data into model inputs without leaking the future. Covers scaling, encoding, missing values, time windows, target encoding and class imbalance, and keeping training and serving identical.
Models eat numbers. Real data arrives as timestamps, categories, nested JSON, missing fields and duplicate users. Feature engineering turns that mess into a matrix that a linear model or a gradient-boosted tree can learn from. It has to do this without leaking the future, and it has to apply the same transforms in training and in serving.
On tabular problems, features often matter more than the choice between boosting libraries. A better definition of days_since_last_active can beat another round of hyperparameter search.
Concepts in this topic
Read in this order. Each page builds on the one before it.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Scaling and transforms | How do you put numbers on a comparable scale? |
| 2 | Encoding categories and text | How do you turn labels and words into numbers? |
| 3 | Missing values and pipelines | What does a gap mean, and where should the fill be fitted? |
| 4 | Point-in-time features | Could this value have existed at prediction time? |
| 5 | Target encoding without leakage | How do you encode a category with thousands of values? |
| 6 | Class imbalance | What do you do when positives are rare? |
Interactions and a minimal feature set
Linear models cannot learn "if A and B" unless you give them the product or a crossed one-hot. Add a few hand-crossed features that domain experts trust, such as mobile × new_user. Trees find interactions on their own, so spend your time on clean, informative columns. Neural nets often turn wide categorical columns into embeddings.
For many B2B churn or conversion problems, this short list beats a naive rule:
- Activity counts over 1, 7 and 28 days.
- Intensity ratios, such as 7-day over 28-day activity, which show acceleration or decay.
- Time since the last activity, and tenure since signup.
- Plan, seat count or revenue bucket.
- Support tickets over 30 days, with a missing indicator.
Train a logistic model and a boosted tree on only these. If neither beats the heuristic on a time split, more exotic encodings will not save you. Revisit the labels and the decision.
Common mistakes across the topic
- Fitting transforms on all the data. Scalers, imputers and encoders are fitted on the training split only, inside the pipeline.
- Using different code in training and serving. Notebook pandas and a rewritten production SQL query drift apart. Share one implementation, or serialize the whole pipeline.
- Searching hyperparameters before features. Fix the features first, then tune.
Key takeaways
- Scale and transform numbers for linear models. Trees care less about scale and more about missing values and categories.
- Use one-hot or hashing for categories by default. Target-encode high-cardinality fields only with out-of-fold statistics.
- Treat missingness as a feature, and impute inside a pipeline fitted on training data.
- Every window aggregate must be correct as of the prediction time.
- Ship the transforms as one pipeline shared by training and serving.
Further learning
A practical, example-driven book on preparing predictors:
Feature Engineering and Selection (Kuhn and Johnson)Feature Engineering and Selection: A Practical Approach for Predictive Modelsfeat.engineeringFor features, training-serving skew and data flows in production:
Designing Machine Learning Systems (Chip Huyen)oreilly.comThe scikit-learn preprocessing guide documents the APIs used on these pages:
scikit-learn User Guide: PreprocessingThe sklearn.preprocessing package provides several common utility functions and transformer classes to change raw feature vectors into a representation that is more suitable for the downstream esti...scikit-learn
Related
Evaluating models
Choose metrics that match the decision, set thresholds from costs, check calibration, and guard against leakage. A high offline score means little until it holds up on real decisions.
Trees and gradient boosting
Decision trees, random forests and XGBoost/LightGBM: still the best default for tabular data.
ML in production
Feature stores, training-serving skew, monitoring for drift and the lifecycle of a deployed model.