Overfitting and curve-fitting (the cardinal sin)
Overfitting is tuning a strategy to past noise instead of real signal. Learn how curve-fitting happens, why a perfect backtest is a red flag, and how to spot it.
Part of the Backtesting and Research track on Agenticks. About 10 minutes, written for a advanced reader.
Every backtester eventually meets the temptation. You have a rule. It is okay, not amazing. So you nudge a setting. The result improves. You nudge another. Better still. A few hours later you are looking at an equity curve that climbs in a near-perfect line, and you feel like you have found something. You probably have not. What you have most likely done is overfit: you tuned the strategy so tightly to one stretch of history that it now describes that exact past instead of any repeatable pattern. This is the cardinal sin of backtesting, and almost everyone commits it at least once.
Markets are signal plus a lot of noise
A price chart is part real, repeatable behavior (signal) and a large part random wiggle that will never happen the same way again (noise). A good strategy latches onto the signal. An overfit one memorizes the noise, which feels like genius on the old data and like nonsense on new data.
Here is the trap stated plainly. Past data contains coincidences: a stop that happened to sit one tick below three random lows, an entry time that lined up with two big moves by luck, a filter that just so happened to dodge the worst week. These are accidents of that particular history. They carry no information about the future. When you keep adjusting rules until the backtest looks perfect, you are not discovering an edge. You are slowly bending the strategy around those accidents. The more knobs you turn, the more of the past you can reproduce, including the parts that were pure chance. That is overfitting, and its close cousin is curve-fitting: literally shaping the rules to trace the exact curve of one chart.
A suit tailored to one photograph
Imagine tailoring a suit to fit a person frozen in a single photo: arms at that exact angle, weight shifted just so. It will fit that frame flawlessly and look absurd the moment the person moves. An overfit strategy is stitched to one snapshot of the market. The market always moves.
You have two strategies with similar backtest returns. Strategy A uses 2 simple parameters. Strategy B uses 14 finely tuned parameters. Which is more likely to be overfit, and why? Strategy B, because each extra parameter is another knob that can be bent to fit past noise Correct. Every added parameter gives the strategy another degree of freedom to reproduce the exact past, including its coincidences. More knobs make a beautiful backtest cheaper to manufacture.
Overfitting leaves fingerprints. Once you know them, you start spotting overfit strategies in seconds: An almost-perfect equity curve. Real edges have ugly stretches. A line that climbs with barely any drawdown is a red flag, not a trophy. A pile of parameters, each tuned to an oddly specific value like a stop of 11 ticks or an entry only between 9:47 and 9:52. Fragility. Change one setting by a hair and the whole result falls apart. A robust edge survives small changes; an overfit one shatters. A collapse on new data. The strategy that looked flawless on the years you built it on does nothing, or loses, on years it never saw. That last point is the real test, and it has a name. Out-of-sample data is history the strategy was never tuned on. If an edge is real, it should at least partly survive there. If it evaporates, you were fitting noise.
- Smells overfit
- A near-perfect equity curve with almost no drawdown, Fourteen parameters, each tuned to a very specific value, Results that collapse when one setting changes slightly
- More trustworthy
- Two simple rules that still work on data they were never tuned on, A messy but positive equity curve over many trades
- Overfitting
- Tuning a strategy so tightly it captures noise, not signal
- Curve-fitting
- Shaping the rules to trace one chart's exact twists
- Parameter
- An adjustable setting, and another knob that can be bent
- Out-of-sample data
- History the strategy was never tuned on
None of this means you can never adjust a strategy. Tuning is normal. The danger is optimization done blindly: running a computer to search thousands of parameter combinations and keeping whichever set produced the best backtest. That process is almost guaranteed to find the settings that fit past noise the best, because with enough combinations some of them will look brilliant by pure luck. The defense is not to stop tuning. It is to tune on one slice of history and then judge the result on a different slice the rules never touched. Build on what is called in-sample data, then check on out-of-sample data. If the edge holds up on data it has never seen, you have something worth more study. If it falls apart, you were fitting the curve, and the perfect backtest was a mirage.
Put these steps in the order that protects you from overfitting.
- Write a simple rule with as few parameters as you can
- Build and tune it on one slice of history (in-sample)
- Check the result on data the rules never touched (out-of-sample)
- Distrust the result if it only shines on the data it was built on
noise signal parameters out-of-sample
A results screen shows the whole ride, including drawdown. An honest edge looks bumpy here; a suspiciously smooth, near-perfect curve is a sign the rules were fit to one exact past.
The instinct that gets you into trouble is a good one: you want the strategy to be better, so you keep improving it. The fix is to point that effort somewhere safer. Improve the idea, not the fit. Ask why a rule should work before you ask how to make its backtest prettier. When a result looks too clean, treat your own suspicion as a feature. The best backtesters are a little disappointed by a perfect curve, because they know what it usually means.
You can now spot a curve-fit
You know that overfitting means fitting noise instead of signal, that more parameters make it easier, and that out-of-sample data is the honest check.
Common questions
- What is overfitting in a trading backtest?
- Overfitting is when a strategy is tuned so tightly to past data that it captures noise and coincidences instead of a repeatable pattern. It looks excellent on the data it was built from and then falls apart on data it has never seen.
- What is the difference between overfitting and curve-fitting?
- They are closely related. Curve-fitting is shaping a strategy's rules and settings to match the exact twists of a historical chart; overfitting is the general result, a model that has memorized the past instead of learning something that generalizes. In practice people use the terms almost interchangeably.
- Why is a near-perfect backtest a warning sign?
- Real edges are messy and survive losing stretches. A backtest with almost no drawdown and a near-perfect equity curve usually means the rules were bent to fit every wiggle of one specific stretch of history, which rarely repeats.
- How do you guard against overfitting?
- Keep the rules simple, use few parameters, and check the result on out-of-sample data the rules were never tuned on. If performance collapses on unseen data, the edge was probably fitted to noise.
Terms defined in this lesson
Continue
Sources
- Bailey, D. H., Borwein, J., Lopez de Prado, M., & Zhu, Q. J. (2014). Pseudo-mathematics and financial charlatanism: The effects of backtest overfitting on out-of-sample performance. Notices of the American Mathematical Society, 61(5), 458-471.
- Harvey, C. R., & Liu, Y. (2015). Backtesting. The Journal of Portfolio Management, 42(1), 13-28.