Q1 · ML basics
What is the bias-variance tradeoff?
Bias is the error that stays when a model is too simple to capture the pattern (it underfits). Variance is the error that comes from a model changing a lot when you change the training sample (it overfits). Total expected error is the sum of bias, variance and irreducible noise. You want a model that is flexible enough to fit the signal, but stable enough that a new sample of the same data does not swing its predictions around.
In simple words: A model that is too stiff misses the pattern; a model that is too wiggly memorizes the sample you happened to get.
Why it works
Think of predicting a student's end-of-term mark from hours of study. A straight line through a clearly curved pattern is biased: it is wrong in a systematic way, even if you collect more students. A very flexible curve that ducks through every training point has low bias on that sample, but if you draw a new class of students the curve jumps. That jumpiness is variance.
The usual picture is a U-shaped test-error curve. As you add capacity (a more flexible model, more features, fewer constraints), bias falls and variance rises. The best test error sits in the middle. Regularization, more data, and ensembles are three different ways of moving that middle.
The snippet below fits a straight line and a high-degree polynomial to the same noisy curve, then reports training and test mean squared error (MSE). MSE is the average of the squared gaps between prediction and true value.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error
rng = np.random.default_rng(0)
X = np.linspace(0, 1, 40).reshape(-1, 1)
y = np.sin(2 * np.pi * X[:, 0]) + rng.normal(0, 0.25, size=40)
Xtr, Xte, ytr, yte = X[:30], X[30:], y[:30], y[30:]
for deg in (1, 9):
model = make_pipeline(PolynomialFeatures(deg), LinearRegression())
model.fit(Xtr, ytr)
tr = mean_squared_error(ytr, model.predict(Xtr))
te = mean_squared_error(yte, model.predict(Xte))
print(f"degree {deg}: train MSE {tr:.3f}, test MSE {te:.3f}")Output, real run on Python 3.13.5:
degree 1: train MSE 0.183, test MSE 0.604
degree 9: train MSE 0.022, test MSE 49603.677The degree-1 line is too simple: both errors stay fairly high. The degree-9 curve almost memorizes the 30 training points (tiny train MSE) and then fails on the held-out points (huge test MSE). That is high variance in numbers, not in a slogan.
Common mistakes
- Saying "bias is underfitting and variance is overfitting" and stopping. Those are the usual symptoms, but the definitions are about systematic error versus sensitivity to the training sample.
- Thinking a more complex model is always better once training error is low. Look at test error.
- Treating the tradeoff as a law that more data cannot change. More data usually lowers variance, so you can afford a more flexible model.
Follow-ups
- Can a model have high bias and high variance at once? Yes. A messy, poorly specified model can be both systematically wrong and unstable.
- Where does bagging help? Bagging (training the same kind of model on many bootstrap samples and averaging) mainly cuts variance. That is why random forests work well with deep trees.
- What is irreducible error? (Advanced) Noise that no model can remove, because the same inputs can map to different outcomes in the real world.