How to take a crap on quantitative backtesting.

With the efforts I have been gaining to show up some misconceptions about investing in local stocks, I would have thought that I should be become by now public enemy number 1 from value investors. Sadly, I have read some attempts to discredit quantitative backtesting and I find that they are either holding back key information, or generally lack intellectual sophistication.

As such, I will take on the burden of criticising quantitative backtesting myself. This is because, of all people, I cannot afford to drink my own Kool-aid. Not only could I stand to lose money if my methodology is flawed, it can affect the portfolios of my students too.

There are, in fact, three valid responses when faced with quantitative backtesting data.

The first, which I support, is to simply accept that is a superior approach towards investing. The second is to disagree and argue that you can get superior returns because the model inadequately capture all the risks associated with this strategy. ( For example, the strategy of small stocks is plagued with the problem of illiquidity. ) The third is to simply argue that backtest models model the past and the strategy simply will not work in the future. You need to have a better way to make money if you argue as such.

A good quantitative strategist must address three specific issues :

a) Data Mining

If you spend enough time interrogating data, you will always find a combination of factors that can result in excellent past performance. There are hundreds of factors to play with, all the quant needs to do is to set a time range and start grinding the numbers through the factors and pick out the best one of the lot. The problem is always the question of whether the strategy works in the future.

The only way I address the problem of data mining is to use only tested factors in the US and repeat the experiment in SGX. Then I do some work to ensure that I'm not crazy by testing out the strategy in Bursa Malaysia. I can safely say that dividends sustainable by free cash flow has not only passed all the tests but also receive support from value investors.

b) Non-stationarity

Imagine you have a bag of red and white balls. You draw out the balls one by one. Over time you have a certain idea of the proportion of red balls to white balls after a while. Sadly the markets do not work like this - when you invest in the markets, the markets will change the bag every now and then. So when you are having the Great Recession and lowering interest rates, you are drawing from a different bad than when you are facing the trade war between China and the US.

This is a much harder issue to address as this means that the returns and standard deviation can change as we enter a new stage of history.

This is not a easy issue to address. My only defence is that if there is a significant change in history, it should be argued as such. We had the same PAP government for 50 years. If anything, Singapore investors have a greater reason to stick to models captured in the past than, say, Turkish investors.

My only defence is to keep expanding the backtest timelines and watch other markets when strategies begin to fail. Of late, the small firm effect is not as profitable as before and value has been losing to growth for over a decade in the US.

c) Model misspecification

This is so hard, I struggle to understand how to even deal with it. When we fail to select factors when we create a model, a factor like the low-variance effect can either be interpreted as a market beating anomaly or a risk premium. So the model is incomplete.Academic researchers have found 316 different factors in quantitative investing.

 The combination of factors I employ are the obvious ones documented by texts. The sample size of Singapore markets is also very small such that using more than 2 factors at one go will result in a set of stocks that may be too small for diversification.

So I have chosen to ignore this critique of quantitative investing.

Maybe a professional quant can share some ideas on how to address these issue when you run a professional fund.