Survivorship bias in backtesting: how current constituent lists hide failures
Survivorship bias makes a backtest trade only the companies that survived. Why a current constituent list hides failures, which way the error points, and what fixes it.
Survivorship bias in backtesting is the error you get when the list of companies a strategy can trade is taken from the present. A constituent list downloaded today names the companies that survived until today. A simulation over that list never holds a company that went bankrupt, was bought out, or was removed from its exchange. Those are the names that screens for cheap, distressed or volatile stocks tend to pick up. For a long-only strategy that would have held such names, the direction is known: the missing losses make returns come out too high, and because the missing names are usually the volatile ones, measured drawdown and volatility are usually flattered too.
A current list answers the wrong question
Index membership is a time series, not a set. A constituent file answers "who is in this index today?" A backtest needs "who was in this index on 9 March 2009?" The difference is the set of companies removed in between: dropped for shrinking, acquired, or bankrupt. Over a decade the two lists diverge substantially, and the names that left are not a random sample.
Vendor price files fail the same way by a different route. The listing file is a snapshot of today. Price history is then pulled backwards for the tickers in it, and a delisting calendar is attached where the vendor has one. Where the calendar is incomplete, the price series simply stops, and a strategy that bought the name two years earlier is graded as if it had sold at the last published quote.
Two distinct defects hide here. Missing securities: the failed company is absent from the universe altogether. Missing terminal returns: the company is present, but its series ends at the last exchange price rather than at what shareholders finally received. A vendor can fix the first and still have the second.
Which way the error points, and how far
Delisting outcomes are lopsided. Companies acquired for cash or stock usually leave at a premium. Companies removed for cause, meaning bankruptcy, liquidation or failing a listing standard, often lose most of their remaining value, and common shareholders in a bankruptcy frequently receive nothing, though not always. That second group is the left tail of stock returns, and it is concentrated in the small, cheap and volatile names that value and small-cap screens already overweight. The bias is largest for strategies that hold the kind of company that fails, and a strategy whose missing names were mostly acquired at a premium could even be understated.
The best-known measurements come from CRSP, the standard academic price database, whose delisting returns were often missing in the 1990s. Shumway (1997) recovered off-exchange prices for NYSE and AMEX stocks delisted for performance reasons and found an average delisting return of about -30%: roughly -23% from recovered quotes alone, and about -33% once firms he identified as worthless were counted. Shumway and Warther (1999) found an average of about -26% for the Nasdaq delisting returns they could recover, and estimated about -55% after allowing for returns they could not locate and for illiquidity. These are averages for particular exchanges and decades, not a universal haircut.
Why the delisting record is the hard part
Two things are missing from ordinary price data, and neither can be reconstructed from the price file itself.
The final return. The last close on the exchange is not what a shareholder ended up with. After a delisting for cause, the stock often keeps trading off-exchange under a modified ticker, and the position ends at whatever a reorganization or liquidation eventually paid: some cash, shares in a new entity, or nothing. A price series that stops at the last exchange print has quietly set the remaining return to zero, an optimistic assumption for a loss that was still unfolding.
The dates. The announcement, the exchange's formal removal filing, the last trading day and the index removal can be weeks apart. A simulation can only exit at a price that was actually available; once trading stopped, the position has to be carried to its payout or its successor security rather than assumed sold. And it may act on the announcement only from the day it was public. Acting earlier is the timing error described under look-ahead bias.
What point-in-time membership fixes
Membership becomes a row per security per index with a validity interval and a separate knowledge date: this security was a member from 2004-05-03 until 2011-03-31. Rebuilding the universe for any past date is then a query.
universe(index = "sp500", as_of = "2009-03-09")
-> every security whose membership interval covers 2009-03-09,
including those whose membership ended later
Nothing is deleted when a membership ends, so failed companies stay in the database and appear in every past universe for which they were eligible. The same discipline has to reach prices and identifiers, as described in point-in-time data.
A worked example
Ten small-cap stocks, bought in equal amounts on 2007-01-02 and held without rebalancing to 2011-12-30. The universe is built once as it stood in 2007, and once as it would be rebuilt from a current constituent file, which keeps the seven securities still listed today. The figures are invented to make the arithmetic easy to follow; they are not measurements.
| 2007-01-02 to 2011-12-30 | Universe as of 2007 | Universe from today's list |
|---|---|---|
| Securities held | 10 | 7 |
| Weight per position | 10% | 14.3% |
| Cumulative return | +18% | +60% |
| Annualized return | 3.4% | 9.9% |
| Annualized volatility | 34% | 26% |
| Maximum drawdown | -58% | -41% |
| Return divided by volatility | 0.10 | 0.38 |
The three missing securities returned -100%, -88% and -52%, a total of -240 percentage points. The seven survivors averaged +60%, a total of +420. Across all ten that is +180 divided by 10, or +18%. One of the ten was acquired in a stock deal, so its return continues in the acquirer's shares, which still trade. The survivors' portfolio is the same seven holdings at larger weights, one seventh each instead of one tenth. The volatility and drawdown rows are illustrative; a real portfolio could move differently.
What "survivorship-bias-free" has to mean
The phrase is claimed far more often than it is defined. Four properties have to hold at once.
| Question to ask | Failure if the answer is no |
|---|---|
| Does the database retain securities delisted before the query date, and include them in a past universe only when they were listed and eligible then? | Failures deleted, or dead names traded |
| Does every delisted security carry a return to its final payout? | Remaining loss recorded as zero |
| Is membership stored as dated intervals keyed by a stable security id? | Ticker reuse merges unrelated companies |
| Does the price series continue where the stock kept trading off-exchange? | Tail cut off at a fake exit price |
A vendor that passes the first test and fails the second has moved the bias rather than removed it. That combination is common, because a delisting calendar is easier to license than delisting returns.
What Quantverse serves today is narrower than this list. The databank API resolves which security a ticker named on a given date, using the SEC's recorded ticker tenures, and keeps a renamed security's history in one lineage that is served as currently known; the catalog entry is CIK, CUSIP and FIGI mappings. The listings, delistings and halts package that would carry the terminal events is listed as coming soon. Tier contents are on the pricing page.