Variance, luck, and separating skill from noise
Variance can disguise luck as skill and skill as failure. Learn how sample size, expectancy, and a range of outcomes help separate a real edge from noise.
Part of the Backtesting and Research track on Agenticks. About 10 minutes, written for a advanced reader.
Two traders run the same rule for a month. One finishes up a lot and feels brilliant. The other finishes down and feels broken. Here is the uncomfortable part: it can be the same rule, with the same true edge, and the difference between them is mostly the order the trades happened to fall in. That scatter has a name. Variance is how much results bounce around their long-run average from one stretch to the next. A coin that lands heads half the time will still hand you five heads in a row sometimes. A trading rule with a small genuine edge will still hand you a losing week, then a winning week, then a flat one, in no tidy pattern. The edge is the average you would reach over thousands of trades. Variance is everything that happens on the bumpy road there. The danger is that variance is a brilliant disguise. A lucky run looks identical to skill while it is happening. A real edge going through a rough patch looks identical to a broken strategy. You cannot feel the difference from the inside, which is exactly why so many people quit good rules and marry bad ones.
Variance is the noise sitting on top of the signal
Signal is the repeatable edge: the average result you would reach over a huge number of trades. Noise is the random scatter on top of it from one stretch to the next. Over a few trades the noise is louder than the signal, so the result you see is mostly variance, not truth.
This is the heart of the luck versus skill question. Luck versus skill is simply asking whether a result came from a repeatable edge or from random chance. The honest answer over a short run is almost always, we cannot tell yet. Walk through why. Suppose a rule truly wins 55 percent of the time with wins and losses of equal size. That is a real edge. Run it for ten trades and you might see seven wins (looks amazing), or you might see four wins (looks broken). Both are completely normal for a 55 percent rule over ten tries. The true edge did not change between those two runs. Only the dice did. Now flip it. A rule with no edge at all, a pure coin flip, will also produce a seven-win run sometimes. So a great short streak is consistent with a strong rule, a weak rule, and no rule. That is what it means to say luck dominates small samples: the result you got is compatible with too many different truths to single one out. This is why people fool themselves in both directions. They take a lucky streak as proof of genius and bet bigger right before variance reverts. Or they take an unlucky streak as proof of failure and abandon a rule that was fine, right before it would have recovered.
luck sample expectancy noise
So how do you actually separate the two? You cannot remove variance, but you can shrink how much it can fool you. Three levers do most of the work. Sample size. The more trades behind a number, the less room luck has to dominate it. Ten trades tell you almost nothing. A few hundred, read honestly, start to mean something. This is the single biggest lever, and the one people respect the least. Expectancy over win rate. Expectancy is the average you would expect to win or lose per trade across the whole sample, blending how often you win with how much you win or lose. A streak can flatter a win rate. Expectancy measured over a large sample is much harder for noise to fake. A range, not a point. One backtest is one ordering of your trades. Reshuffle the same wins and losses and the equity curve changes shape and depth. A Monte Carlo simulation does exactly this, reordering the same trades thousands of ways to show the spread of outcomes luck alone could have produced. If your one result sits near the lucky edge of that spread, your edge is thinner than the single line suggested. Notice what none of these give you: certainty. They give you better odds of not being tricked. That is the whole game. You are never proving skill beyond doubt, you are lowering the chance that you mistook noise for it.
- Sample size
- Limits how much luck can dominate a result
- Expectancy
- The average result per trade over the whole sample
- Monte Carlo simulation
- Shows the range of outcomes luck could have produced
- Variance
- The random scatter you are trying to see past
Judge a result against the range it could have been
A single profitable backtest is one path out of many. The honest question is not whether this path made money, but where it sits among all the paths the same edge could have produced. A result that needed a lucky ordering to look good is a result you should not trust much.
Reordering the same trades thousands of ways turns one equity curve into a fan of outcomes. Your single backtest is just one line inside that range.
A rule has a real edge and wins 55% of trades with equal win and loss sizes. After 12 trades it has lost money. What is the most honest read? Twelve trades is far too small to tell luck from a real edge; the result is mostly noise Correct. A 55 percent rule losing over 12 trades is completely normal variance. Over a small sample, luck dominates and the outcome cannot single out the truth.
Order these from least to most convincing as evidence that an edge is real, not luck.
- One profitable month
- A profitable backtest over a few hundred trades
- That backtest checked with a Monte Carlo range of reshuffled outcomes
- The same edge holding up on new data it was never built on
There is a mindset that follows from all this, and it is mostly humility. If you accept that variance never disappears, a few habits fall out on their own. You stop reading a single week as a verdict. You size your risk so a normal losing run, the kind a real edge produces, cannot take you out before the edge has room to show. You hold judgement until the sample is large enough to mean something, and you treat a lucky-looking streak with the same suspicion as an unlucky one. None of this promises an outcome. A strategy with a real, tested edge can still lose for a long time, and a person who understands variance perfectly can still have a hard year. Understanding luck and skill does not remove luck. It just stops you from drawing the loud, wrong conclusion that variance is always whispering: that the last few trades were the whole truth.
You can now see past the noise to the signal
You know that variance disguises luck as skill and skill as failure, and that sample size, expectancy, and a range of outcomes are how you separate a real edge from chance.
Common questions
- What is variance in trading?
- Variance is how much results scatter around their long-run average from one stretch to the next. Even a strategy with a real edge will have winning runs and losing runs purely from the order trades happen to fall in. Variance is the noise that sits on top of the signal.
- How do you tell luck from skill?
- You cannot tell from a single run. Over a small number of trades, luck dominates and a good streak can look exactly like skill. Only a large sample, read for expectancy and the worst losing stretch, starts to reveal whether a repeatable edge is really there.
- Why does a profitable backtest still not prove skill?
- Because one backtest is one path out of many that could have happened. Reorder the same wins and losses and the equity curve changes shape. A single profitable line can be a lucky ordering of an edge that is weak or even absent.
Terms defined in this lesson
Continue
Sources