Backtesting Overview and Biases

Some important topics to keep in mind as one develops and backtests a trading strategy are:

Look-Ahead Bias: Bias created by the use of information or data in a study or simulation that would not have been known or available during the period being analyzed. This will usually lead to inaccurate results in the study or simulation.  An example of this would be: having a strategy enter on a bar based on an indicator reading, but the indicator is actually available after the close of that bar.

Sampling Bias: A type of bias caused by choosing non-random data for statistical analysis. The bias exists due to a flaw in the sample selection process, where a subset of the data is systematically excluded due to a particular attribute. The exclusion of the subset can influence the statistical significance of the test, or produce distorted results.  An example of this would be testing a strategy on the stocks of only one sector, or testing a momentum strategy during a time of momentum in the market.

Live Fill Price != Backtested Fill Price: When a strategy is developed assuming the fills that happened in the backtest are also ones that it is able to get, and with sufficient liquidity .  An example of this would be backtesting fills within the top 5% of a trading range. 

Stationarity: Commonly called "de-trending" in time series analysis, may leave out important information through the fit of the raw data to a stochastic process.  For example, liquidity is at least partially dependent on price levels, and de-trending ignores this.

Sample Size: Having a large enough sample of observations (trades) to perform statistical analysis.  An example of this is analyzing a strategy that makes 20 trades a year.

Solutions

To address these issues, I will be using a couple of methods.  First, through Esignal, I can backtest on 1-minute bars over 120 trading days.  For initial backtests and analysis, I will be using time periods of 10-30 days.  I will be testing strategies that have at least 50 trades over the test window.  I will be using indicators and metrics that are only available at or before the time of the entry, and with fills that are conservatively obtained.  Also, I will be testing the strategy over 20 stocks (biased towards liquid stocks however), namely: BHI, RIG, MS, GS, JPM, WFC, X, NUE, POT, AGU, UNP, NSC, AFL, MET, AAPL, RIMM, SPY, OIH, XLF, XME. 

As the strategy I am working on goes through progressive versions, I will backtest on different time periods as well to confirm that the modifications are satisfactory (e.g.: V1 for t1-t2, V2 for t1-t2 & t2-t3).  Lastly, as I get closer to a viable strategy that can run all the time on real money, I will be running it through suitable resampling procedures (e.g.: bootstrapping, jacknifing, and Monte Carlo permutation tests).  I believe that strategies that survive this will be very robust, and those are the ones I will take live.