Model Training and Evaluation: Loss, Overfitting, and Accuracy

When people train their first model, they often focus only on accuracy. To understand whether a model is reliable, you need to know what training adjusts, what a loss function measures, why overfitting happens, and why test data must stay separate.

This article explains the basics of model training and evaluation: parameters, loss functions, epochs, overfitting, validation data, test data, and common classification metrics.

If the previous article explained how to organize a machine learning project, this one explains how to decide whether the training process is trustworthy.

1. What Does Training Adjust?

A model can be viewed as a function with parameters:

prediction = model(input_features, parameters)

Before training, parameters may be random or initialized with default values. Training adjusts those parameters so model output becomes closer to the true labels.

A very simple linear model looks like this:

y = w1 * x1 + w2 * x2 + b

Here, w1, w2, and b are parameters. Training tries to find better values for them.

2. What Is a Loss Function?

The model needs a way to measure how wrong a prediction is. That measurement is the loss function.

For regression, a simple loss can be squared error:

loss = (y_true - y_pred) ** 2

For classification, cross-entropy loss is common. You do not need to derive the formula at the beginning, but the intuition matters:

A confidently wrong prediction receives a large loss. A prediction close to the correct answer receives a smaller loss.

During training, the algorithm tries to reduce the overall loss.

3. The Intuition Behind Gradient Descent

Many models use gradient descent or a variant of it to update parameters. Think of it as walking downhill:

1. The current parameters produce a loss value
2. The algorithm estimates which direction reduces loss
3. The parameters move a small step in that direction
4. The process repeats many times

An important hyperparameter is the learning rate. If it is too small, training is slow. If it is too large, training can bounce around or fail to converge.

new_weight = old_weight - learning_rate * gradient

This is not the full mathematical story, but it explains why training loops repeat parameter updates.

4. Epoch, Batch, and Iteration

Deep learning training often uses these terms:

• Epoch: one full pass through the training set
• Batch: a small group of samples used for one update step
• Iteration: one parameter update

If the training set has 1000 samples and the batch size is 100, one epoch contains 10 iterations.

Traditional machine learning libraries may not expose these terms directly, but the basic idea is similar: the model uses training data to adjust parameters.

5. Why Overfitting Happens

Overfitting means the model performs well on training data but much worse on new data.

Common causes include:

• The model is complex enough to memorize noise and details in the training set
• The training data is too small to represent the real problem
• The features contain information that should not be available, also called data leakage
• The model trains for too long without validation monitoring

The danger is that training metrics can look excellent while real-world performance is poor.

6. Training, Validation, and Test Data

For reliable evaluation, data is often split into three parts:

• Training set: used to fit parameters
• Validation set: used to tune settings, select models, and watch for overfitting
• Test set: used at the end to estimate final generalization

For small practice projects, a training/test split can be enough. But remember: the test set should not be used repeatedly for tuning, or it becomes part of the decision process.

7. Common Classification Metrics

Classification should not be judged by accuracy alone. These metrics often appear together:

• Accuracy: the proportion of correct predictions
• Precision: among predicted positives, how many are truly positive
• Recall: among true positives, how many were found
• F1-score: a combined measure of precision and recall

For medical screening, missing a real positive case may be costly, so recall may matter more. For automatic account blocking, falsely blocking normal users may be costly, so precision may matter more.

8. Reading a Confusion Matrix

A confusion matrix compares predicted labels with true labels:

                  predicted negative  predicted positive
true negative              TN                  FP
true positive              FN                  TP

• TP: a positive sample predicted correctly
• TN: a negative sample predicted correctly
• FP: a negative sample incorrectly predicted as positive
• FN: a positive sample incorrectly predicted as negative

The advantage of a confusion matrix is that it shows not only how many mistakes happened, but also which direction those mistakes went.

9. Evaluation Checklist

When evaluating a model, check these questions:

• Was the test set isolated from training?
• Are the classes heavily imbalanced?
• Did you look beyond accuracy?
• Was the model compared with a simple baseline?
• Did you inspect some wrong predictions manually?
• Is the gap between training performance and test performance too large?

The point of training is not merely to push one metric upward. The point is to build a trustworthy evaluation process and understand when the model is likely to fail.

10. Evaluation Evidence Matrix

A reliable training report connects metrics to decisions. The table below shows what to record before deciding that a model is ready for a demo, article, or deployment experiment.

Evidence item	What to record	Decision it supports	Warning sign
Split integrity	Random seed, stratification, grouping rule, and test isolation	Whether the test score estimates new-data behavior	Repeatedly tuning on the test set until the score improves
Metric choice	Accuracy, precision, recall, F1, ROC-AUC, or task-specific cost	Which error type matters most	Using accuracy on a heavily imbalanced dataset without context
Baseline comparison	Majority-class baseline, simple linear model, or previous version	Whether the trained model adds value	A complex model beats no documented baseline
Error review	Confusion matrix and representative wrong predictions	Whether failures are random, systematic, or unacceptable	High average score hides rare but costly mistakes

11. What a Trustworthy Training Record Includes

A useful training record should include at least these details:

• How training, validation, and test data were split
• The model, important parameters, and random seed
• Training metrics and test metrics, not just one final score
• Error analysis, especially for the most costly error types
• Comparison against a simple baseline model

These notes may look small, but they make the experiment auditable when you return to it later.

Use the validation set enough and it becomes a second training set

The distinction between training, validation and test sets above has a consequence worth stating separately, because it explains why models that were “tuned for ages with great validation scores” so often fail on real data.

Model parameters are learned from the training set — that much is explicit. But hyperparameters — learning rate, depth, regularisation strength, feature combinations — are learned from the validation set. Every cycle of “check the validation score, change a setting, check again” uses validation-set information to update your choices. Do that a few hundred times and you are fitting the model to the validation set using a very crude optimiser.

The validation score then stops meaning what you want it to mean. It no longer answers “how does this model perform on unseen data” but “how does this model perform on this one specific validation set” — which you have now been optimising against for hundreds of rounds.

The standard remedy is a third split you never look at:

training     learns parameters      -- used directly by the model
validation   selects hyperparams    -- you inspect it repeatedly, so it gets fitted
test         final report           -- sealed throughout, opened once

The discipline is in “opened once.” If you look at the test score, dislike it, and go back to tuning, that test set is spent — it has become a second validation set. The honest response at that point is to carve out fresh test data, or to report the current result along with an accurate account of the tuning that preceded it.

Three splits are a luxury when data is scarce, and cross-validation replaces a fixed validation set there: divide the training data into k folds, validate on each in turn, and average. Every record contributes to training, and the validation score becomes more stable — a single split’s score fluctuates considerably on small data, and reading one number invites being misled by randomness.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_train, y_train, cv=5)
print(scores.mean(), scores.std())   # the standard deviation matters equally

Read the standard deviation alongside the mean. The two sets of five-fold scores below have identical means of 0.896:

A: 0.91  0.89  0.90  0.90  0.88     mean 0.896   std 0.010
B: 0.98  0.76  0.97  0.78  0.99     mean 0.896   std 0.103

Reported by mean alone, these two models look identical. But A is stable while B is acutely sensitive to how the data was divided — same model, same data, and a different split swings the result from 0.76 to 0.99.

Faced with B, the right move is not to report 0.896 but to find out why certain folds are so much worse. Common causes: a class with few samples that a random split can drop entirely into one fold; or grouping structure in the data (several records from one user, several samples from one batch) split across training and validation, causing leakage. Both call for stratified or grouped splitting rather than accepting that mean.

12. What to Read Next

The previous article is Machine Learning Workflow. After training and evaluation are clear, continue with Neural Network Basics to connect parameters with multi-layer function composition.