Overfitting in Machine Learning: What It Is and How to Avoid It
A model that fits your training data perfectly can be worthless—because it has learned the noise, not the signal. Overfitting is the central pitfall of predictive modelling, and understanding it is what separates a model that works from one that only looks like it does.
There is a seductive but dangerous milestone in predictive modelling: the moment your model predicts the training data almost perfectly. It feels like success. Often it is the opposite. A model that reproduces its training data flawlessly has frequently learned not the genuine pattern but the random noise particular to that sample—and when it meets new data, it fails. This is overfitting, and it is the single most important concept to understand in machine learning and predictive research, because nearly every serious modelling mistake is a version of it.
This guide explains what overfitting is, why it happens, how to detect it, and the standard techniques for avoiding it. It builds on our guide to prediction vs explanation and reflects the evaluation discipline in our Machine Learning, AI & Text-as-Data practice.
What overfitting is
Every dataset contains two things: the true underlying pattern (the signal) and random, sample-specific variation (the noise). A good model captures the signal and ignores the noise, so that what it learns generalises to new data drawn from the same process. Overfitting is what happens when a model captures the noise as well—fitting the idiosyncrasies of the particular sample it was trained on so closely that it mistakes them for real structure.
The visual intuition is a model that draws a wildly wiggling line to pass through every training point exactly, versus a smooth curve that follows the general trend. The wiggling line has lower error on the training data—it hits every point—but it will predict badly on new data, because the wiggles were fitted to noise that will not recur. The smooth curve has some training error but generalises well. Overfitting is preferring the wiggling line: chasing training accuracy at the expense of the only accuracy that matters, on data the model has not seen.
The mirror image is underfitting: a model too simple to capture even the true signal, which performs poorly everywhere. The goal is the balance between them—complex enough to learn the pattern, simple enough not to learn the noise. This trade-off, often described as the bias–variance trade-off, is the central tension in fitting any predictive model.
Why it happens
Overfitting has recognisable causes. The most common is a model that is too complex for the data—too many parameters, too much flexibility—relative to the amount of information available, giving it the capacity to memorise rather than generalise. A closely related cause is too little data: with few observations, even a moderate model can fit the sample’s quirks, because there is not enough signal to pin down the real pattern. Too many features relative to observations, and training for too long in iterative methods, both push in the same direction. In every case the underlying story is the same—the model has more freedom to fit the data than the data has genuine structure to constrain it.
Impressive training performance is a warning sign, not a success. The only performance that matters is on data the model was not trained on. A near-perfect fit to the training set should raise suspicion of overfitting, not confidence.
How to detect it
Overfitting is detected by comparing performance on data the model was trained on with performance on data it has never seen. The foundational technique is the train–test split: hold out a portion of the data, train on the rest, and evaluate on the held-out portion. A model that performs well on the training set but markedly worse on the test set is overfitting—the gap between training and test performance is the signature of the problem.
More robust is cross-validation, which repeatedly splits the data into training and validation folds, trains and evaluates across the rotations, and averages the results. This gives a more reliable estimate of how the model will perform on new data than a single split, and makes better use of limited data. The essential principle behind all of this is simple and inviolable: never judge a model on the data it was trained on, and never let the test data influence the modelling in any way, or it stops being an honest test.
How to avoid it
Several complementary techniques reduce overfitting. Simplify the model—fewer parameters or features—so it has less capacity to memorise; often a simpler model that generalises beats a complex one that overfits. Get more data where possible, since more observations give the model more genuine signal to learn and less room to fit noise. Regularisation—techniques that penalise excessive model complexity during training—is a standard and powerful tool for keeping a model from fitting too aggressively. Feature selection removes irrelevant predictors that offer only noise to fit. And in iterative methods, early stopping—halting training when validation performance stops improving—prevents the model from continuing to fit noise after it has learned the signal.
None of these is a silver bullet, and the right combination depends on the problem. But they share a philosophy: constrain the model’s freedom to match the data’s genuine information content, and validate honestly on unseen data throughout. Applied together, they turn a model that merely fits into one that predicts.
Why it matters for research
In research, overfitting is not just a technical performance issue—it is a threat to the validity of what you claim. A predictive model reported on its training performance can look impressive and mean nothing, and results that are not validated out-of-sample are, at best, unverified and, at worst, illusory. This is why honest out-of-sample evaluation is the expected standard for any predictive study, and why a knowledgeable reviewer will look for it first. It is also why, when the goal is explanation rather than prediction, an overfit model is doubly misleading—its inflated in-sample fit can lend false confidence to relationships that will not replicate. Guarding against overfitting is, in the end, part of the broader discipline of reporting only what the evidence genuinely supports.
The bottom line
Overfitting is the gap between fitting your data and understanding your problem. A model that memorises the training set’s noise will fail on new data, however impressive its training accuracy looks—so training performance is never the measure of success. Detect overfitting by evaluating on held-out data through a train–test split or cross-validation, and avoid it by simplifying the model, gathering more data, regularising, selecting features, and stopping training at the right point. The whole aim is a model that generalises: one whose performance on data it has never seen is the performance you can actually rely on.
Frequently asked questions
Want a model that actually generalises?
From regularisation and feature selection to rigorous cross-validation, our team builds predictive models that hold up on new data—and reports them to a standard reviewers trust.