Real-world datasets rarely come perfectly formatted for modeling. A well-designed data preprocessing pipeline ensures that you apply the same transformations consistently across training and production environments.
Why a Pipeline Rather Than a Script
The argument for pipelines is not tidiness, it is correctness. Any transformation that learns something from the data, such as a mean for imputation, a scale factor, or a category vocabulary, is a fitted parameter. If it is estimated on the full dataset before splitting, information from the validation and test sets leaks into training, and your estimate of model performance becomes optimistic.
A pipeline enforces the boundary structurally. Fitting happens on training folds only, and the learned parameters are then applied unchanged to held-out data. The same fitted object is what you serialise and deploy, which eliminates the classic failure where preprocessing in production drifts away from preprocessing in the notebook.
Handling Missing Values
Start by assessing the extent of missing data. Common strategies include dropping incomplete rows, filling numeric columns with the mean or median, and using the most frequent category for categorical features.
Before choosing, it is worth asking why the values are missing, because the mechanism determines what is safe:
- Missing completely at random (MCAR). Missingness is unrelated to anything. Dropping rows is unbiased, merely wasteful.
- Missing at random (MAR). Missingness depends on observed variables, for example income being unreported more often in certain regions. Conditional imputation can recover the structure.
- Missing not at random (MNAR). Missingness depends on the unobserved value itself, such as high earners declining to state income. No imputation fixes this; the missingness is informative and should be modelled explicitly.
The practical compromise is to impute with the median for skewed numeric columns and the mode for categoricals, while adding a binary indicator recording that the value was missing. If the missingness carries signal, the model can use it; if not, the indicator costs almost nothing.
Encoding Categorical Variables
Many machine learning algorithms require numeric inputs. Techniques like one-hot encoding or ordinal encoding convert categories into numbers. Scikit-learn’s ColumnTransformer allows you to apply different encoders to different columns in a single pipeline.
Choose based on the variable and the model. One-hot encoding is the safe default for nominal categories, though it becomes unwieldy at high cardinality. Ordinal encoding is correct only when the categories genuinely have an order, such as small, medium, large; applying it to unordered categories invents a numeric relationship the model will happily exploit. Tree-based models tolerate ordinal encoding of nominal variables better than linear models do, since trees can split the ordering into arbitrary groups.
Two details cause production failures. Unseen categories at prediction time will raise errors unless the encoder is configured to ignore them, so set handle_unknown="ignore". And target encoding, which replaces a category with the mean outcome for that category, leaks the label directly unless it is computed within cross-validation folds.
Scaling and Normalization
Scaling features to a common range prevents variables with large magnitudes from dominating a model. Standardization (mean of zero, unit variance) is typical for linear models, while min-max scaling keeps values between 0 and 1.
\[z = \frac{x - \mu}{\sigma}, \qquad x_{\text{minmax}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}} .\]Which models actually need it is worth knowing precisely. Distance-based methods such as k-nearest neighbours, k-means, and SVMs with RBF kernels require scaling, because the distance metric is otherwise dominated by whichever feature has the largest units. Gradient descent converges faster on scaled features, which matters for neural networks and for linear models fitted iteratively. Regularised regression needs it because the penalty applies equally to all coefficients, so unscaled features are penalised inconsistently.
Decision trees, random forests, and gradient boosting do not need scaling at all, since they split on thresholds and are invariant to monotone transformations. Scaling them wastes effort without causing harm.
When outliers are present, RobustScaler centres on the median and scales by the interquartile range, so extreme values do not distort the parameters used for everything else.
Putting It All Together
Use scikit-learn’s Pipeline to chain preprocessing steps with your model. This approach guarantees that the exact same transformations are applied when predicting on new data, reducing the risk of data leakage and improving reproducibility.
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
numeric_features = ["age", "income", "tenure_months"]
categorical_features = ["region", "plan_type"]
numeric_pipe = Pipeline([
("impute", SimpleImputer(strategy="median", add_indicator=True)),
("scale", StandardScaler()),
])
categorical_pipe = Pipeline([
("impute", SimpleImputer(strategy="most_frequent")),
("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])
preprocess = ColumnTransformer([
("num", numeric_pipe, numeric_features),
("cat", categorical_pipe, categorical_features),
])
model = Pipeline([
("prep", preprocess),
("clf", RandomForestClassifier(n_estimators=300, random_state=0)),
])
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(f"ROC AUC: {scores.mean():.3f} +/- {scores.std():.3f}")
Because the whole object is passed to cross_val_score, imputation statistics, category vocabularies, and scaling parameters are refitted inside every fold. That is what makes the reported score an honest estimate rather than a lower bound on optimism.
Deploying the Same Object
Serialise the fitted pipeline rather than the model alone:
import joblib
model.fit(X_train, y_train)
joblib.dump(model, "churn_model.joblib")
# in the serving process
loaded = joblib.load("churn_model.joblib")
predictions = loaded.predict_proba(new_records)[:, 1]
The serving code now takes raw records in the original schema. There is no separate preprocessing implementation to keep in sync, which removes an entire category of training-serving skew.
Two operational cautions. Pickled objects are tied to library versions, so pin your dependencies and record the versions alongside the artefact. And a pipeline validates nothing about incoming data: pair it with an explicit schema check so that a renamed column or a unit change fails loudly instead of silently producing predictions from garbage.
References
- Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830.
- Kuhn, M., & Johnson, K. (2019). Feature Engineering and Selection: A Practical Approach for Predictive Models. CRC Press.
- van Buuren, S. (2018). Flexible Imputation of Missing Data (2nd ed.). CRC Press.
- Sculley, D., et al. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28.
