How do I tell whether my model is underfitting or overfitting?

Compare training error to test error. If both are high and close together, the model cannot even fit the data it has seen, which is bias, and a more expressive model or better features is the answer. If training error is low and test error is much higher, the model memorised the training set, which is variance, and regularisation, more data, or a simpler model is the answer. The gap between the two curves is the diagnostic, not either number alone.

Is the tradeoff always a tradeoff?

Not always. Adding training data reduces variance without adding bias, so it improves both sides at once, which is why it is worth exhausting before tuning anything else. Better features can do the same, giving the model a more learnable representation without making it more complex. The strict tradeoff bites when you are adjusting model complexity with a fixed dataset, which is the usual situation but not the only one.

Which techniques target which side?

Regularisation, ensembling by averaging like bagging, and simplifying the model all attack variance. More features, a more expressive architecture, and less aggressive regularisation attack bias. Cross-validation does not fix either; it measures them honestly, which matters because a single train and test split can make a high-variance model look fine by luck.

Why does high variance show up as different predictions on different training sets?

Because the model is fitting patterns that exist in one particular sample rather than in the underlying process. Noise differs between samples by definition, so a model that fitted the noise produces different answers when the noise changes. That instability is what variance measures, and it is why evaluating on a single held-out set can mislead: you are seeing one draw from a distribution of possible models.

Can a model have both high bias and high variance?

Yes, and it is more common than the tidy diagrams suggest. A poorly specified model with the wrong features can be simultaneously unable to capture the real relationship and unstable across samples, because it is latching onto weak accidental patterns instead. When both training error is high and it moves a lot between splits, the problem is usually the feature representation rather than the model class.

Journal AI Integration January 26, 2023

AI Integration · Machine Learning

Bias vs Variance: The Key to Successful Predictive Modeling

Understand the bias-variance tradeoff in machine learning with mathematical formulas, visual explanations, and strategies to find the right balance.

Anurag Verma

4 min read

Bias-Variance Machine Learning Overfitting Statistics Model Selection

Bias vs Variance: The Key to Successful Predictive Modeling

Key takeaways

Bias is systematic error: the model is consistently wrong in the same direction, because it is too simple to represent the real relationship.
Variance is sensitivity to the training set: retrain on a different sample and the predictions move, because the model fitted noise as if it were signal.
High bias with low variance is underfitting. Low bias with high variance is overfitting. Both produce poor test error, for opposite reasons, which is why test error alone does not tell you which you have.
Model complexity is the dial that trades one for the other. More features and parameters cut bias and raise variance; simpler models do the reverse.
More training data is the exception that improves both, which is why it is the first thing to try before reaching for regularisation or a different architecture.
The diagnostic is the gap between training and test error. Both high means bias; training low and test high means variance.

As a machine learning and data science student, you’ve probably heard the terms bias and variance thrown around quite a bit. But what do these terms actually mean, and why are they so important? In this post, we’ll take a closer look at bias and variance, and discuss how to balance them for optimal performance in your models.

What bias is

Bias refers to the difference between the predicted values of a model and the true values of the data. In simpler terms, it’s the degree to which a model’s predictions are consistently incorrect. For example, imagine you’re trying to predict the price of a car based on its features. A model with high bias might always predict the price to be lower than it actually is, regardless of the specific car.

What variance is

On the other hand, variance refers to the variability of a model’s predictions for different training sets. In other words, it’s the degree to which a model’s predictions change depending on the specific data it’s trained on. For example, imagine you’re using the same car price prediction model, but you train it on two different datasets. A model with high variance might give you very different predictions for the same car depending on which dataset it was trained on.

Why the balance matters

So why is it important to balance bias and variance? A model with high bias and low variance is said to be underfitting the data, meaning it’s not capturing the complexity of the relationship between the input and output variables. On the other hand, a model with low bias and high variance is said to be overfitting the data, meaning it’s fitting the noise in the training data rather than the underlying pattern. The goal is to find a model that has a good balance of bias and variance, known as good fit.

Techniques for finding a good fit

There are several techniques you can use to achieve good fit. Cross-validation, regularization, and ensemble methods are some of the popular methods. Another way to balance bias and variance is through the use of different model architectures and hyperparameter tuning. For example, using a more complex model with more features and parameters can decrease bias but increase variance, while using a simpler model with fewer features and parameters can decrease variance but increase bias.

It’s important to remember that bias and variance are not always independent, and in some cases, reducing one may also reduce the other. For example, increasing the amount of training data can reduce both bias and variance.

The maths behind both

First, let’s start with the mathematical equations. In a linear regression model, the equation for predicting a continuous target variable, y, based on a single input variable, x, is:

y = mx + b + ε

where m is the slope of the line, b is the y-intercept, and ε is the error term.

Bias refers to the difference between the predicted values of the model and the true values of the data. We can express this mathematically as:

Bias = E[(mx + b) - y]

where E[ ] denotes the expected value.

Variance, on the other hand, refers to the variability of a model’s predictions for different training sets. We can express this mathematically as:

Variance = E[(mx + b)^2] - (E[mx + b])^2

A worked example

Now, let’s take a look at a machine learning example. Imagine we are trying to predict the price of a house based on its square footage. We train a linear regression model using a dataset of 100 houses. The model has a low bias and high variance, meaning it fits the training data well, but it doesn’t generalize well to new data. When we test the model on a new dataset of 50 houses, we find that its predictions are far off from the true prices.

In this example, our model is overfitting the training data. To improve its performance, we can try to reduce the variance by using a simpler model, such as a linear regression with regularization, or by increasing the amount of training data.

The short version

In summary, bias and variance are two key concepts in machine learning and data science that describe the errors that can occur in model predictions. Bias refers to the difference between the predicted values of a model and the true values of the data, and variance refers to the variability of a model’s predictions for different training sets. Balancing these errors matters for achieving good performance and finding a model that generalizes well to new data.

Frequently asked questions

How do I tell whether my model is underfitting or overfitting?: Compare training error to test error. If both are high and close together, the model cannot even fit the data it has seen, which is bias, and a more expressive model or better features is the answer. If training error is low and test error is much higher, the model memorised the training set, which is variance, and regularisation, more data, or a simpler model is the answer. The gap between the two curves is the diagnostic, not either number alone.
Is the tradeoff always a tradeoff?: Not always. Adding training data reduces variance without adding bias, so it improves both sides at once, which is why it is worth exhausting before tuning anything else. Better features can do the same, giving the model a more learnable representation without making it more complex. The strict tradeoff bites when you are adjusting model complexity with a fixed dataset, which is the usual situation but not the only one.
Which techniques target which side?: Regularisation, ensembling by averaging like bagging, and simplifying the model all attack variance. More features, a more expressive architecture, and less aggressive regularisation attack bias. Cross-validation does not fix either; it measures them honestly, which matters because a single train and test split can make a high-variance model look fine by luck.
Why does high variance show up as different predictions on different training sets?: Because the model is fitting patterns that exist in one particular sample rather than in the underlying process. Noise differs between samples by definition, so a model that fitted the noise produces different answers when the noise changes. That instability is what variance measures, and it is why evaluating on a single held-out set can mislead: you are seeing one draw from a distribution of possible models.
Can a model have both high bias and high variance?: Yes, and it is more common than the tidy diagrams suggest. A poorly specified model with the wrong features can be simultaneously unable to capture the real relationship and unstable across samples, because it is latching onto weak accidental patterns instead. When both training error is high and it moves a lot between splits, the problem is usually the feature representation rather than the model class.

Newer dispatch

Evaluation Metrics for Classification and Regression

AI Integration

Older dispatch

Predicting Medical Costs with Multivariate Linear Regression