As a follow-up to yesterday’s post, I wanted to provide some more background about back-testing. As usual, I won’t re-invent the wheel, so let me present some great posts from others. QuantStart gives an overview of back-testing, including the key reasons why back-testing is important (filtration, modeling, optimization, and verification) as well as the biases that create misleading back-testing results (optimization bias, look-ahead bias, survivorship bias, and psychological tolerance bias). and finally the pros and cons of some of the software tools available for back-testing (Excel / OpenOffice; Matlab / Octave / SciLab; Python / Ruby / Erlang / Haskell; R / SPSS / Stata; and C++ / C# / Java / Scala). The whole article is here.
I find QuantStart’s point about psychological tolerance bias particularly relevant. At sites like Collective2, I routinely see outraged posts from subscribers to other systems that they are forced to endure 3% drawdowns while trading highly leveraged systems that have CAGRs of 100%+. It’s easy to agree to hypothetical risks, but when it’s real cash on the line, many find they don’t have the stomach for drawdowns. The solution is position sizing–if the risk is too high, reduce the amount of money you have at risk. We’ll revisit this topic later.
Meanwhile, via Bloodhound System comes another article on back-testing from The Algorithmic Trader, who elaborates on some of the dangers inherent in back-testing, including allowing your system to accidentally see future data, survivorship bias, curve-fitting, and compounding. As always, read it all. The Algorithmic Trader has several great articles on algorithmic trading for beginners, so please visit his site for more. I will be highlighting several of his points in future posts as well.