Why Sample Size Matters in Trading Strategy Testing
Sample size is simply how many trades your test is based on, and it matters because small samples lie. With only ten or twenty trades, luck dominates the result, so a losing strategy can look brilliant and a good one can look broken, purely by chance. The more trades you have, the more the random noise averages out and the closer your measured win rate and expectancy get to the truth. There's no magic number, but a few dozen trades is rarely enough, a few hundred starts to mean something, and more is better as long as the trades are recent enough to still reflect how markets behave. Sample size also applies across conditions: two hundred trades all from one calm bull market still tell you little about how the strategy handles a crash.
How to think about trade sample size when evaluating a strategy's statistical validity and why small samples produce unreliable conclusions.
Key points
- Sample size is the number of trades behind a test result, and small samples are dominated by luck.
- With few trades, a bad strategy can look great and a good one can look terrible, so the numbers can't be trusted.
- As trade count grows, random noise averages out and the measured stats move closer to the real edge.
- A few dozen trades is usually too few, while a few hundred begins to carry real meaning.
- Sample size also means variety, so hundreds of trades from one market mood still don't show how a strategy handles other conditions.
- Fast strategies gather large samples quickly, while a strategy that trades a few times a month needs years to build a trustworthy record.
Frequently asked questions
How many trades do I need for a reliable backtest?
There's no exact threshold, but a few dozen is rarely enough and several hundred is a reasonable place to start trusting the numbers. The goal is enough trades that one or two lucky or unlucky results don't swing the overall picture.
Why do small sample sizes give misleading results?
Because luck has an outsized effect on small numbers. Flip a coin ten times and you might get eight heads, which looks like a bias but isn't. The same thing happens with trades, so a short test can show a fake edge or hide a real one.
Does more data always mean a better backtest?
More trades usually help, but only if they're still relevant. Data from decades ago may reflect a market that no longer behaves the same way. You want a large sample that also stays recent enough and spans different conditions.
How does sample size relate to expectancy?
Expectancy is an average, and averages need many data points to settle. On a small sample, the expectancy figure bounces around and can easily be positive by chance even for a losing strategy, which is why sample size and expectancy go hand in hand.
How do I get a big enough sample to trust?
You can ask the AlgoAgent to backtest a strategy over long histories so it generates a large number of trades across different market conditions, then report how many trades the result rests on, so you know whether the sample is big enough to mean something.
Related on Agenticks
This content is for educational purposes only and does not constitute financial advice. Trading involves risk, including possible loss of capital.