In many machine learning projects the model itself is not what decides the result first. What usually separates outcomes is feature engineering: how missing values are handled, how categorical variables are encoded, whether numeric features are standardised, and whether any data leaks between the training and test sets.
This article walks through a small feature engineering pipeline using scikit-learn’s Pipeline and ColumnTransformer. By the end you will be able to put missing-value handling, categorical variables and numeric standardisation into one reusable training flow.
If you are still deciding which model to pick, read How to Choose a Machine Learning Algorithm first. This article addresses the other frequent question: how to actually do feature engineering.
1. What feature engineering is
Feature engineering turns raw data into input a model can learn from reliably. It is not simply “cleaning the data”; it is making the structure of the data suit the model.
- Missing-value handling: missing age, missing income, unknown category
- Categorical encoding: turning text categories such as city, major or device type into a form the model can read
- Numeric scaling: putting numeric features with different units onto a comparable scale
- Feature combination: combining existing fields into new fields that carry more meaning
- Leakage prevention: test set information must not enter the statistics computed during training
The same model can behave completely differently depending on its input features. The goal of feature engineering is not to manufacture complicated fields, but to preserve genuinely useful information and keep the training process reproducible.
2. Why not process the full dataset before splitting
Many beginners run mean imputation, standardisation and categorical encoding over the full dataset, and only then split into training and test sets. This is a common data leakage risk.
The correct approach is to learn the fill values, scaling parameters and encoding rules on the training set only, then apply that same set of rules to the test set. scikit-learn’s Pipeline is designed for exactly this flow.
3. Preparing a mixed feature dataset
The small example dataset below simulates real tabular data. It contains numeric features, categorical features and missing values at the same time.
import pandas as pd
data = pd.DataFrame({
"study_hours": [2.0, 5.5, 1.0, 7.0, None, 3.5, 6.0, 8.0],
"sleep_hours": [6.0, 7.5, None, 8.0, 5.0, 6.5, 7.0, 8.5],
"major": ["CS", "Math", "Art", "CS", "Business", "Math", None, "CS"],
"uses_planner": ["yes", "yes", "no", "yes", "no", "no", "yes", None],
"passed": [0, 1, 0, 1, 0, 0, 1, 1],
})
X = data.drop(columns="passed")
y = data["passed"]This example does not represent real educational assessment; it exists only to demonstrate the machine learning feature-processing flow. In a real project, avoid feeding sensitive variables, biased variables or non-compliant data straight into a model.
4. Handling numeric and categorical features separately with ColumnTransformer
Numeric and categorical features cannot use the same treatment. Numeric features suit median imputation and standardisation; categorical features suit most-frequent imputation and one-hot encoding.
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
numeric_features = ["study_hours", "sleep_hours"]
categorical_features = ["major", "uses_planner"]
numeric_transformer = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
])
categorical_transformer = Pipeline([
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore")),
])
preprocessor = ColumnTransformer([
("num", numeric_transformer, numeric_features),
("cat", categorical_transformer, categorical_features),
])handle_unknown="ignore" matters here. It means that if a category the encoder never saw during training appears in the test set, the encoder will not raise an error but will represent it with an all-zero one-hot vector.
5. Putting feature engineering and the model in one Pipeline
Next, connect the preprocessor to the model. Training, prediction and cross-validation then all use the same processing flow automatically.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
model = Pipeline([
("preprocess", preprocessor),
("classifier", LogisticRegression()),
])
scores = cross_val_score(model, X, y, cv=4, scoring="accuracy")
print("CV accuracy:", scores)
print("Mean accuracy:", scores.mean())In a real project you can swap LogisticRegression for a random forest, gradient boosted trees or another model, while the preprocessing structure above stays reusable.
6. Complete runnable code
Here is the integrated version, which you can save directly as feature_engineering_demo.py and run.
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
data = pd.DataFrame({
"study_hours": [2.0, 5.5, 1.0, 7.0, None, 3.5, 6.0, 8.0],
"sleep_hours": [6.0, 7.5, None, 8.0, 5.0, 6.5, 7.0, 8.5],
"major": ["CS", "Math", "Art", "CS", "Business", "Math", None, "CS"],
"uses_planner": ["yes", "yes", "no", "yes", "no", "no", "yes", None],
"passed": [0, 1, 0, 1, 0, 0, 1, 1],
})
X = data.drop(columns="passed")
y = data["passed"]
numeric_features = ["study_hours", "sleep_hours"]
categorical_features = ["major", "uses_planner"]
numeric_transformer = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
])
categorical_transformer = Pipeline([
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore")),
])
preprocessor = ColumnTransformer([
("num", numeric_transformer, numeric_features),
("cat", categorical_transformer, categorical_features),
])
model = Pipeline([
("preprocess", preprocessor),
("classifier", LogisticRegression()),
])
scores = cross_val_score(model, X, y, cv=4, scoring="accuracy")
print("CV accuracy:", scores)
print("Mean accuracy:", scores.mean())Run it with:
python3 -m venv .venv
source .venv/bin/activate
pip install scikit-learn pandas
python feature_engineering_demo.py7. Common feature engineering mistakes
- Processing the full dataset before splitting: easily leaks test set information into the training stage
- Numbering categorical variables directly: encoding cities as 1, 2, 3 can make the model read an ordering that does not exist
- Ignoring differences between training and production data: without
handle_unknown, a new category in production can make the prediction flow fail - Dropping too many incomplete samples: missingness itself may carry information, and deleting it changes the data distribution
- Not saving the preprocessing flow: whatever happens during training must happen identically at prediction time
8. Which matters more, feature engineering or hyperparameter tuning
If the data is rough, do feature engineering first; once the data structure is reasonable, move on to tuning. In many projects, adding correct missing-value handling, categorical encoding and a leak-free flow is more effective than swapping models blindly.
A practical order:
- Start with a reproducible data split.
- Fix the preprocessing flow with a Pipeline.
- Train a simple baseline model.
- Inspect the misclassified samples and the important features.
- Only then compare different models and parameters.
9. Feature engineering audit table
The place feature engineering usually goes wrong is not whether the code runs, but whether training, validation and production inference all use the same traceable set of rules. The table below works as a checklist for any tabular machine learning project.
| Stage | What to check | Recommended practice | Failure signal |
|---|---|---|---|
| Split order | Whether the train/validation/test split happens before fitting the imputer, scaler and encoder | Put every preprocessing step inside Pipeline and ColumnTransformer |
Means, variances or category sets computed on the full dataset before the test split |
| Categorical variables | Whether new, rare and missing categories have an explicit strategy | Use handle_unknown="ignore" and record the rule for merging rare categories |
The prediction flow fails outright when a new city, device or major appears in production |
| Numeric scaling | Which models depend on scale and which do not | Standardise for linear models, KNN, SVM and neural networks; keep tree models as a control | Model output changes noticeably after a unit change, with no explanation in the training record |
| Reproducibility | Whether the preprocessing object, model version, field order and training random seed are saved | Serialise the full pipeline and write unit tests for the input schema | The training script and the prediction service each carry their own preprocessing logic, with mismatched field order |
10. FAQ
Do all numeric features need standardising?
Linear models, KNN, SVM and neural networks generally need it more. Tree models are insensitive to feature scale, though standardising them is not always harmful — what matters is keeping training and prediction consistent.
Does one-hot encoding create too many features?
With a small number of categories, one-hot encoding is usually fine. If a field has thousands or tens of thousands of categories, consider merging rare categories, target encoding, hashing tricks or another representation.
Should missing values be filled with the mean or the median?
When the numeric distribution is skewed or contains many outliers, the median is usually more stable. Categorical features are commonly filled with the most frequent category, or given their own Unknown value.
11. What to read next
Feature engineering lets a model see more stable information, but it does not solve everything. If the training score is strong and the validation score drops noticeably, the next thing to read is How to Fix Overfitting and Underfitting.