How do I diagnose which one I have?

Plot training and validation error together as training progresses. Both high and close means underfitting: the model cannot fit even what it has seen, so it needs more capacity or better features. Training error low with validation error much higher means overfitting: it memorised the training set. Validation error falling and then rising while training error keeps falling is the classic overfitting curve, and the turning point is where early stopping should fire.

Which fix should I try first?

More or better data, if you can get it, because it reduces overfitting without adding bias, unlike every other option here. After that, early stopping, since it is nearly free to implement and catches the most common failure. Regularisation next. Reach for a different architecture last, because changing the model class means re-tuning everything downstream and it is rarely where the actual problem was.

Is data augmentation always useful?

No, it depends on whether you can transform an example without changing its label. For images this is easy: a rotated, flipped or slightly recoloured cat is still a cat, so you can multiply the dataset cheaply. For tabular data it is much harder, because there is no transformation of a row of customer attributes that obviously preserves meaning. Synthetic approaches like SMOTE exist but can create examples that are not realistic, so they need checking rather than trusting.

How does regularisation actually prevent overfitting?

By adding a penalty proportional to the size of the model's weights to the loss it is minimising. That means the model can no longer reduce training error for free; every increase in complexity has to earn its place against the penalty. L2 shrinks weights toward zero smoothly, L1 drives some of them exactly to zero and so performs feature selection as a side effect. The strength of the penalty is itself a hyperparameter to validate, since too much regularisation turns overfitting into underfitting.

Why isn't cross-validation a solution by itself?

Because it is a measurement, not an intervention. What it gives you is a more reliable estimate of generalisation error, by evaluating on several different splits instead of trusting one. That reliability matters most exactly when a model has high variance, because a single split of a high-variance model can look excellent by accident. But knowing accurately that you are overfitting does not change the model; regularisation, more data, or a simpler architecture does.

Journal AI Integration February 16, 2023

AI Integration · Machine Learning

Overfitting and Underfitting: Causes and Fixes

Understand overfitting and underfitting in ML models with practical solutions including cross-validation, regularization, early stopping, and data augmentation.

Anurag Verma

4 min read

Overfitting Underfitting Machine Learning Regularization Cross-Validation Model Selection

Overfitting and Underfitting in Machine Learning

Key takeaways

Overfitting is high training accuracy with poor test accuracy: the model learned the noise. Underfitting is poor accuracy on both: the model never learned the signal.
The usual cause of overfitting is too many parameters for the amount of data, or training for too many iterations past the point of diminishing return.
Underfitting comes from the opposite: too few parameters for the complexity of the problem, or stopping training too early.
Regularisation penalises complexity in the loss function, which pushes the model toward simpler weights and better generalisation.
Early stopping halts training when validation performance stops improving, which is the cheapest defence against learning noise and costs nothing to add.
Data augmentation attacks overfitting by manufacturing more training examples from the ones you have, which works well for images and poorly for tabular data.
Cross-validation does not fix either problem. It measures them honestly, which matters because a single train/test split can make an unstable model look fine by luck.

Machine learning algorithms are used to make predictions, identify patterns and trends, and classify data based on past examples. However, to create models that accurately predict future data, it’s essential to balance the amount of information that a model can use to learn with the amount of information that it can generalize. Overfitting and underfitting are two common problems that occur when training machine learning models.

Overfitting

Overfitting occurs when a model is too complex, and it has learned the training data too well, to the point that it becomes overly specific to the training data and is unable to generalize to new data. This results in a model that is very accurate on the training data, but performs poorly on new, unseen data.

One of the most common causes of overfitting is when the model has too many features or parameters relative to the size of the training dataset. When this occurs, the model may fit the training data too closely and fail to generalize to new data. Another cause of overfitting is when the model is trained for too many iterations, causing it to learn the noise in the training data rather than the underlying patterns.

For example, let’s consider a model that is trained to predict whether a patient has diabetes based on several features, such as their age, BMI, blood pressure, and glucose levels. If the model is too complex, it may learn patterns that are specific to the training data, such as the fact that patients with a particular age range and blood pressure are more likely to have diabetes. However, this may not generalize to new data, where the patterns may be different.

Underfitting

Underfitting occurs when a model is too simple and is unable to capture the underlying patterns in the training data. This results in a model that is not accurate on either the training data or new data.

One of the most common causes of underfitting is when the model has too few features or parameters relative to the complexity of the problem. When this occurs, the model may not be able to capture the underlying patterns in the data. Another cause of underfitting is when the model is not trained for long enough or with enough data, causing it to miss important patterns.

For example, let’s consider a model that is trained to predict housing prices based on the size of the house, the number of bedrooms, and the age of the house. If the model is too simple, it may not capture the complex relationship between these features and the price of the house, resulting in a model that is inaccurate.

Solutions to Overfitting and Underfitting

There are several techniques that can be used to address overfitting and underfitting in machine learning:

Regularization: Regularization is a technique used to prevent overfitting by adding a penalty term to the loss function of the model. This penalty term encourages the model to use simpler weights and biases, which in turn helps it to generalize better to new data.
Cross-validation: Cross-validation is a technique used to evaluate a model’s performance on multiple subsets of the data. By testing the model on different subsets of the data, it is possible to identify whether the model is overfitting or underfitting.
Early stopping: Early stopping is a technique used to prevent overfitting by stopping the training process when the model’s performance on a validation set stops improving. This helps to prevent the model from learning the noise in the training data.
Data augmentation: Data augmentation is a technique used to increase the amount of training data by creating new examples through transformations of the existing data. This can help to prevent overfitting by giving the model more examples to learn from.
Feature selection: Feature selection is a technique used to select the most important features from a dataset and remove any irrelevant features. This helps to reduce the complexity of the model and prevent overfitting.
Ensemble methods: Ensemble methods involve combining multiple models to improve their performance. By using multiple models with different strengths and weaknesses, it is possible to create a more robust model that can handle both overfitting and underfitting.

Conclusion

Overfitting and underfitting are common problems in machine learning, but there are several techniques that can be used to address them. By using regularization, cross-validation, early stopping, data augmentation, feature selection, and ensemble methods, it is possible to create models that are more accurate and better able to generalize to new data. Understanding these concepts and techniques is essential for anyone working in the field of machine learning, and can help to ensure that their models are as accurate and effective as possible.

Frequently asked questions

How do I diagnose which one I have?: Plot training and validation error together as training progresses. Both high and close means underfitting: the model cannot fit even what it has seen, so it needs more capacity or better features. Training error low with validation error much higher means overfitting: it memorised the training set. Validation error falling and then rising while training error keeps falling is the classic overfitting curve, and the turning point is where early stopping should fire.
Which fix should I try first?: More or better data, if you can get it, because it reduces overfitting without adding bias, unlike every other option here. After that, early stopping, since it is nearly free to implement and catches the most common failure. Regularisation next. Reach for a different architecture last, because changing the model class means re-tuning everything downstream and it is rarely where the actual problem was.
Is data augmentation always useful?: No, it depends on whether you can transform an example without changing its label. For images this is easy: a rotated, flipped or slightly recoloured cat is still a cat, so you can multiply the dataset cheaply. For tabular data it is much harder, because there is no transformation of a row of customer attributes that obviously preserves meaning. Synthetic approaches like SMOTE exist but can create examples that are not realistic, so they need checking rather than trusting.
How does regularisation actually prevent overfitting?: By adding a penalty proportional to the size of the model's weights to the loss it is minimising. That means the model can no longer reduce training error for free; every increase in complexity has to earn its place against the penalty. L2 shrinks weights toward zero smoothly, L1 drives some of them exactly to zero and so performs feature selection as a side effect. The strength of the penalty is itself a hyperparameter to validate, since too much regularisation turns overfitting into underfitting.
Why isn't cross-validation a solution by itself?: Because it is a measurement, not an intervention. What it gives you is a more reliable estimate of generalisation error, by evaluating on several different splits instead of trusting one. That reliability matters most exactly when a model has high variance, because a single split of a high-variance model can look excellent by accident. But knowing accurately that you are overfitting does not change the model; regularisation, more data, or a simpler architecture does.

Newer dispatch

Variables and Data Types in Python: A Beginner's Guide

Technology

Older dispatch

EDA With Scatter Plots and Lines of Best Fit

AI Integration

Join the conversation.

Comments are powered by GitHub Discussions. Sign in with your GitHub account to leave a comment.