Why most backtests lie, and how to catch it

A backtest shows what a set of exact rules would have done in the past. It is the best evidence a trader can get before risking money, and it is easy to get wrong in a way that flatters the strategy. Almost every mistake below makes results look better.

Testing on today's stocks

If you test a breakout strategy on the stocks listed today, every company that collapsed, merged or was delisted along the way is missing. Those are exactly the trades that would have hurt. This is survivorship bias, and it lifts results. The fix is a list of what was tradable on each day of the test.

Using information from the future

Look-ahead bias enters through small details. A rule that needs the day's closing price cannot have bought at that day's open. An index membership list from today was not known five years ago. Fundamentals are published weeks after the quarter they describe. Each one lets the backtest use information the trader never had.

Prices you could not have got

Many backtests assume perfect fills. In real trading the entry slips on a breakout everyone is chasing, a stop is filled at the open when the stock gaps through it, and a big order in a small stock moves the price. Assume some slippage, fill gapped stops at the open, and keep positions small next to a stock's daily traded value.

Leaving out the costs

In India STT on a delivery trade is 0.1% on the buy and 0.1% on the sale, before stamp duty, exchange charges, GST and the DP charge. A strategy that makes 0.5% a trade before costs can make far less after them, and a frequent trader can lose the whole edge. The charges calculator shows the break-even move for any trade.

One stock at a time

Testing each stock separately ignores the days when twenty signals arrive and the account can afford three, and the weeks when every open position falls together. Only a test of the whole account, with its cash limit, shows the real drawdown.

Tuning until it works

Try enough settings and one of them will look brilliant by chance. Two habits guard against it. Keep the number of settings small, and check that nearby values also work. If a 20-day lookback is great and 18 and 22 are poor, the result is noise. Then hold back a stretch of history you never tune on, and test it once at the end.

Too few trades

Twenty trades cannot tell skill from luck. The strategy odds simulator shows how widely results spread even when the rules are identical. Aim for a hundred trades or more, across a crash and a sideways market as well as a rally.

The last test

The final check is to run the rules on the live market with no money at stake, and compare what happens with what the backtest says should have happened on the same days. If the two disagree, the backtest assumed something that doesn't happen in live trading.