In-Sample vs Out-of-Sample: The Backtest Split That Stops Overfitting
In-sample data is the price history you use to build and fine-tune your strategy. Out-of-sample data is a separate slice you lock away and never touch until the end, so you can test the finished strategy on prices it has never seen. It's the difference between grading yourself on a practice test you already memorized and sitting the real exam. The single biggest backtesting mistake is judging a strategy on the exact same data you built it on. That almost always looks great and means almost nothing, because you can tune any set of rules to fit the past. Splitting your history into an in-sample build set and an untouched out-of-sample check set is the simplest way to catch a strategy that only works in hindsight.
In-sample data is what you build a strategy on; out-of-sample is the untouched slice you test it on. Here's how that one split catches overfitting before it fools you.
Key points
- In-sample is the data you build on; out-of-sample is the data you hide away and test on only once, at the very end.
- Overfitting is when your rules are tuned so tightly to past price wiggles that they fit that one stretch of history and nothing else.
- A common split is roughly 70% of your history for building and 30% held back, kept in time order so the out-of-sample slice comes after the build period, not shuffled in.
- The out-of-sample test only counts if you look at it once. Peeking, tweaking, and re-testing quietly turns your check set into more build data.
- If a strategy shines in-sample but collapses out-of-sample, that gap is telling you the edge was mostly curve-fitting, not something real.
- In a results screen like Agenticks's Backtest Review, you can see the in-sample and out-of-sample stats side by side instead of trusting a single number.
Frequently asked questions
What is the difference between in-sample and out-of-sample data?
In-sample is the chunk of price history you use to build and adjust your strategy. Out-of-sample is a separate chunk you set aside and don't touch until the strategy is finished, then test on once. The point is to see how the rules behave on data they were never shaped around.
How much data should I use for out-of-sample testing?
A common starting point is around 70% of your history for building and 30% held back for the out-of-sample check, though 80/20 is also fine. Keep it in time order so the held-back slice sits after the build period, not scattered through it. The exact ratio matters less than having enough trades in each part to mean something.
Why does testing on the same data cause overfitting?
Because you can tune almost any set of rules to fit prices that already happened. If you keep adjusting settings until the backtest looks perfect, you're really just memorizing that one stretch of the past. A strategy fit that tightly usually falls apart the moment it meets new prices.
Is out-of-sample testing the same as walk-forward analysis?
They're related but not identical. Out-of-sample testing usually means one build set and one held-back check set. Walk-forward analysis rolls that idea forward again and again, re-building on one window and testing on the next across the whole history, which is a tougher version of the same principle.
Where can I see in-sample vs out-of-sample results?
You want a results screen that keeps the two slices separate instead of blending everything into one score. In Agenticks, Backtest Review shows in-sample and out-of-sample metrics side by side, along with the equity curve, drawdown, and walk-forward, so you can spot a strategy that only worked in hindsight. Treat it as context for how fragile a strategy might be, not a promise about the future.
Related on Agenticks
This content is for educational purposes only and does not constitute financial advice. Trading involves risk, including possible loss of capital.