Survivorship bias: why testing only the winners lies

Survivorship bias flatters results by leaving out funds and stocks that failed. See a worked example, what the research found, and how to test around it.

Part of the Backtesting and Research track on Agenticks. About 11 minutes, written for a advanced reader.

Read first: What is backtesting, Look-ahead bias: when a backtest sees the future.

Last reviewed .

Risk notice. Backtested and hypothetical results are simulated: no trades were placed, the rules were applied to past data with the benefit of hindsight, and they may not reflect liquidity, slippage or every cost. Past results, real or simulated, do not predict future results.

Key takeaways

Say you want to know how stock-picking funds have done over the last twenty years. You download every fund that exists today, look up its twenty-year record, and average them. The answer looks healthy. It is also wrong, and in a predictable direction. The funds that exist today are the ones that survived. Funds that did badly were often closed or merged into other funds along the way, and their records left the list with them. Averaging only the survivors is like judging a marathon by interviewing the people at the finish line. Survivorship bias is that error: drawing conclusions from the cases still around while the ones that dropped out are quietly missing. It matters for backtesting because most data starts from today and looks backwards. Unless someone deliberately kept the failures, they are not there to be tested.

The losers left the dataset

Things rarely disappear at random. Funds close after poor years and companies delist after collapses. When the failures leave and the winners stay, any average of what is left is tilted upward before a single rule is tested.

This is not a theoretical worry. It has been measured. A study that tracked every US mutual fund alive at the end of 1976, including the ones that later merged away, found that funds tend to disappear because of poor performance, and showed how much that distorts results that only look at survivors. A later study of US diversified equity funds from 1962 to 1995 found the bias grows with the length of the sample: from about 0.07% a year for one-year samples to about 1% a year for samples longer than fifteen years. One percent a year sounds small until you remember it compounds, and that it is often about the size of the edge people are looking for. Stocks have the same problem. Research on a widely used US stock database found that correct final returns were missing for most stocks delisted for bad reasons such as bankruptcy, and that those missing returns were large. And failure is common. A study of US stocks since 1926 found that most individual stocks had lifetime buy-and-hold returns below one-month Treasury bills, while the best-performing 4% of companies accounted for the net gain of the whole market. Past returns do not guarantee future returns, but the lesson for testing is clear: a list built from today's names is a list of the lucky minority.

Worked example: ten companies, two answers

Put $100 into each of ten companies at the start of a test. Ten years later, three have been delisted.

Start (10 companies x $100)$1,000
3 delisted companies, final values ($30 left from $300)$10, $0 and $20
7 survivors ($1,050 from $700)$150 each on average
Full list ((1,050 + 30) / 1,000)$1,080, or +8%
Survivors only (1,050 / 700)$1,050 on $700, or +50%
Survivors-only test vs what happened+50% vs +8%

Illustrative numbers, not real companies or a forecast.

In a backtest, survivorship bias usually arrives through one of these doors. Today's index list, used for the past. A rule tested on the current members of a major index back to 2005 only ever trades companies that were going to be big enough to be in it now. The ones that shrank and were dropped are missing. This is also a form of look-ahead bias, because the list itself comes from the future. Picking the famous winner. Testing a breakout rule on a stock because it rose tenfold over the last decade means the hardest part, choosing the winner, was done with hindsight. Datasets that drop dead names. Some price files remove a symbol the day it stops trading. Delisted stocks, closed funds and crypto coins that faded to nothing simply vanish, along with their final losses. Strategies and traders. The strategies people still talk about, and the traders still posting results, are the ones that survived. The ones that blew up went quiet. That is why a winning screenshot proves almost nothing.

There is a quieter version that happens on your own desk. Suppose you test twenty ideas, throw away the eighteen that did badly, and keep the two that looked good. Those two are survivors too. On their own they look impressive, but they were selected from a pile in which most ideas failed, and some of their shine is the luck that got them picked. The fix is the same habit as everywhere else: keep a record of everything you tried, not just what you kept, and judge the survivors against the whole pile. If two out of twenty random ideas would be expected to look this good by chance, two good-looking results are not yet evidence of anything. This is the same trap described in overfitting and curve-fitting, seen from a different angle.

Survivorship risk
Testing a stock-picking rule on today's index members back to 2005, Studying this year's top-rated funds over the past 20 years, Choosing a stock to test because it rose tenfold last decade, Copying the method of a trader who has had five great years
Much lower risk
Using a dataset that keeps delisted stocks and their final returns, Building the list of eligible stocks as it stood on each past date

Defending against survivorship bias comes down to keeping the failures in view. Use data that includes the dead. Survivorship-free data keeps delisted stocks, closed funds and their final returns. It costs more and is harder to find, which is exactly why so many tests skip it. Build the universe as of each date. If a rule trades "the 500 largest companies", work out who the 500 largest were on each past date, using point-in-time data, not today's ranking. Choose before you look. Decide which markets or symbols a rule will be tested on before checking how they performed. If you only test the instruments whose charts look good, you have already filtered for survivors. Say when you cannot. When a dataset genuinely cannot include failures, treat the result as an upper bound, the best the rule could plausibly have done, and expect reality to be lower. Check how much rests on a few names. If one or two stocks produce most of the profit, the result says more about picking them than about the rule. A large, varied sample of trades protects you here too.

A rule tested on today's 30 most valuable companies, back to the year 2000, shows a large gain. What is the main problem? Those 30 were chosen knowing they would become the largest; the companies that faded are missing Right. The list was drawn up at the end of the period, so it contains only the biggest successes. No trader in 2000 knew which 30 these would be.

failures better delisted

You can spot a test that only counts the winners

You know how survivorship bias tilts results upward, what the research on funds and stocks found, and how point-in-time data and keeping delisted names defend against it.

Check your understanding

In the worked example, what did all 10 companies together actually return?
+8%. The full list ended at $1,080 from $1,000, which is +8%. Including the three failures is what makes it honest.
Why does choosing a stock to backtest because it rose tenfold last decade create survivorship bias?
You picked it knowing the outcome, and skipped the many similar stocks that fell. Right. The choice itself used hindsight, so any rule tested on it starts with a winner baked in.
A data provider drops each company on the day it is delisted. What does that do to a stock backtest?
It loses many failed companies' final losses, so results look better than they were. Exactly. Research on US stock data found these missing final returns are large, and they are mostly losses.

Common questions

What is survivorship bias in simple terms?
It is the error of studying only the things that are still around, such as funds still open or companies still listed, and ignoring the ones that disappeared. Because the ones that disappear usually did badly, the survivors make the average look better than it really was.
How does survivorship bias affect a backtest?
If a test picks its stocks from today's list, it already knows which companies made it. Companies that went bankrupt or were removed from an index are missing, along with their losses, so the test reports returns that no trader in the past could have earned.
How do you avoid survivorship bias?
Use data that includes delisted companies and their final returns, build the list of eligible stocks as it stood on each past date, and choose what to test before looking at how it performed. When the data cannot include failures, treat the result as an upper bound.

Terms defined in this lesson

Continue

Sources

Written and reviewed by the Agenticks Learn editorial team. How we write and check lessons.

Agenticks Learn is general education. It is not financial, investment, tax or legal advice, it does not take your situation into account, and nothing on it is a recommendation to buy, sell or hold any security or to open any account. Examples use illustrative numbers. Investing involves risk, including losing money you put in, and past performance does not predict future results. Tax rules and limits change; the official sources linked in each lesson are the final word.