How do I Choose the Best Backtesting Software?

Backtesting software is designed to simulate how well a particular trading strategy would have worked over a specific previous period. The idea is to give some insight into how well the same strategy would work in the future, though by definition this can only be a prediction. Keys to picking the correct backtesting software include avoiding postdictive error, looking for customization options, and avoiding software produced by the same people selling a trading system.

The most fundamental rule of choosing backtesting software is to use packages that allow you to solely use data that would have been available at the time. Not doing this creates a statistical problem known as postdictive error, meaning that the analysis doesn't reflect how a trader would actually have made decisions in carrying out a strategy. One example of this would be if the software worked with closing prices only; this isn't a realistic situation, as by the time that price became available for the hypothetical trader to have made a decision, the market would have closed!

The most accurate way to avoid postdictive error is to carry out the backtesting entirely manually. As this isn't usually practically efficient, it's important to use software that allows as much customization as possible. Generally, the more automated and rigid the software, the more likely it is to include postdictive error.

Another useful way to use backtesting software is to look for applications that make it easy to rerun the analysis with one variable changed. For example, a trader might be planning a strategy that includes selling any stock that has lost 35% of its value. A good application will be able to quickly show what difference would have been made to the results if the trader had instead sold any stock that lost 50% of its value. As well as testing whether the fundamentals of a strategy appear sound, this customization makes it easier to test a strategy against the limitations of human nature. While a trader might believe that the 35% fall is "objectively" the best point at which to sell, he may realize that if he carried out the strategy for real, he would be tempted to let the stock fall further in the hope of a recovery, simply because it can be hard to admit defeat.

Traders should be particularly wary of any backtesting software that is produced by a company that also sells advice on which trading system to use. Partly this is because such companies will be tempted to use a backtesting set-up that is particularly designed to show their system as working well. But even when companies don't act so cynically, it may be the case that the limitations of the backtesting software they have used have influenced their choice of recommended trading strategy.