Once a machine learning model has been trained, the most common confusion is this: the training score is high but the test score is low, or neither the training score nor the test score is any good and no amount of tuning moves them. Those two problems usually correspond to overfitting and underfitting.
This article explains overfitting, underfitting and model tuning with runnable Python examples. By the end you should be able to tell which problem you have from the training and validation scores, and know whether to start with the data, model complexity, regularisation or cross-validation.
If you have just finished Model Training and Evaluation, this is the next step: moving from reading metrics to diagnosing a model.
1. What overfitting and underfitting mean
Overfitting is a model memorising the detail, and even the noise, in the training data. It shows up as a high training score alongside a clearly lower validation or test score.
Underfitting is a model too simple to capture even the main structure in the training data. It shows up as a low training score and a low validation score.
One line to remember it by:
- Training high, validation low: suspect overfitting first
- Training low, validation low: suspect underfitting first
- Training and validation both high: the model is currently in reasonable shape
2. Demonstrating model complexity with a decision tree
A decision tree is well suited to demonstrating overfitting, because the deeper the tree the easier it is to memorise details of the training samples. The code below compares training and validation accuracy across decision trees of different depths.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.25,
random_state=42,
stratify=y,
)
for depth in [1, 2, 3, 4, 6, 10, None]:
model = DecisionTreeClassifier(max_depth=depth, random_state=42)
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(f"max_depth={depth}: train={train_score:.3f}, test={test_score:.3f}")Save it as decision_tree_depth_demo.py and run:
python decision_tree_depth_demo.pyYou will normally see a shallow tree score poorly on training data, which indicates limited expressive power. As the tree gets deeper the training score rises, but the test score does not necessarily keep improving. That is the classic trade-off introduced by model complexity.
3. Telling overfitting apart from underfitting
Do not rely on a single test score. The more reliable diagnosis records the training score and the validation score together.
- Underfitting: training score low, validation score low, so the model has not learned enough structure
- Overfitting: training score high, validation score low, so the model has memorised details of the training data
- Insufficient data: the training score fluctuates and the validation score is unstable, so you may need more samples or cross-validation
- Wrong metric: accuracy is high but the business outcome is poor, which can mean class imbalance or asymmetric error costs
This is exactly why The Complete Machine Learning Workflow stresses splitting into training, validation and test sets. Without reliable evaluation there is no way to tell what has actually gone wrong with the model.
4. How to fix overfitting
The core problem in overfitting is that the model finds it too easy to memorise the training data. The usual directions are:
- Reduce model complexity: limit tree depth, reduce the number of features, use a simpler model
- Add regularisation: increase the regularisation strength on linear models; add weight decay or dropout on neural networks
- Add more data: more samples reduce the chance of the model memorising incidental noise
- Apply data augmentation: especially common in image, text and speech tasks
- Use cross-validation: reduces the randomness of depending on a single split
- Stop early: halt training once validation performance stops improving
For tree models, start by constraining max_depth, min_samples_leaf and min_samples_split. For logistic regression, start with the regularisation parameter C.
5. How to fix underfitting
The core problem in underfitting is that the model lacks expressive power, or the input features do not carry enough information. The usual directions are:
- Move to a stronger model: from a linear model to a tree model, an ensemble or a neural network
- Add meaningful features: bring in domain variables, interaction features or time-window features
- Reduce excessive regularisation: regularisation that is too strong stops the model learning
- Train longer: in deep learning tasks, too few epochs can cause underfitting
- Check label quality: with heavy label noise, a model struggles to learn stable structure
Underfitting is not simply a matter of making the model bigger. If the raw features carry no information, a more complex model can only learn noise.
6. Using cross-validation to avoid misdiagnosis
A single train/test split may happen to land on easy or difficult samples, which makes the judgement unstable. Cross-validation evaluates the model across several different splits, so the result is more reliable.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
for depth in [2, 3, 4, 6, None]:
model = DecisionTreeClassifier(max_depth=depth, random_state=42)
scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
print(f"max_depth={depth}: mean={scores.mean():.3f}, std={scores.std():.3f}")To check the cross-validation result on its own, save this section as tree_cross_validation_demo.py and run:
python tree_cross_validation_demo.pymean represents average performance and std represents the variation between splits. A high mean with large variation means the stability of the model still needs checking.
7. Diagnosis table: identify the problem before tuning
Before tuning anything, look at the training score, the validation score and the cross-validation spread together. The table below helps avoid reading every problem as “the model is not strong enough”.
| Observation | More likely problem | First action | What not to do first |
|---|---|---|---|
| Training high, validation low | Overfitting | Limit complexity, add regularisation, cross-validate | Keep deepening the model |
| Training low, validation low | Underfitting, or features carry no signal | Add effective features, lower regularisation, move to a stronger baseline | Only change the random seed |
| High mean, high std | Unstable splits | Add samples, use stratified sampling, use grouped cross-validation | Report only the best single run |
| Accuracy high, recall low on the key class | Metric mismatch or class imbalance | Inspect the confusion matrix, macro F1 and per-class metrics | Look at overall accuracy alone |
8. A practical tuning order
Do not open with a wide grid search. A steadier tuning order is:
- Confirm first that the data split has no leakage.
- Train a simple baseline model.
- Record the training score and the validation score.
- Decide whether this is underfitting or overfitting.
- Change one class of factor only, such as model complexity or regularisation.
- Use cross-validation to confirm the gain is not incidental.
- Run one final evaluation on an independent test set at the end.
This order is far easier to review afterwards than changing the model, the features, the parameters and the data split at the same time. The thing to fear most in machine learning tuning is an unexplained improvement, because you have no idea whether it will reproduce next time.
9. FAQ
Is 100% training accuracy always a good sign?
Not necessarily. If the validation or test score is clearly lower, 100% training accuracy may instead indicate that the model has overfitted. Read the training score and the validation score together.
Does a low test score always mean the model is not strong enough?
Not necessarily. The score may have dropped back after a data leak was corrected, the label quality may be poor, the training and test distributions may differ, or the evaluation metric may not suit the task.
Should features be removed when overfitting?
It can be worth considering, but do not delete blindly. Check first whether a feature leaks future information, whether it repeats a signal already expressed elsewhere, and whether it only holds in the training data. Keep features that carry business meaning and stay stable.
Can cross-validation replace a test set?
No. Cross-validation suits model selection and tuning; it is still advisable to hold back an independent test set to estimate how the model performs on data that took no part in selection.
10. What to read next
Overfitting and underfitting are the core diagnostic method in machine learning tuning. To continue, return to A Small Python AI Practice Project and apply this diagnosis to a complete scikit-learn classification pipeline.