

Minimizing the "benefit of hindsight" in trading system testing 
Written by Forex Automaton  
Monday, 27 October 2008 16:07  
Once you know something, such as the past history, pretending that you do not is very difficult. The ways a trader can delude him/herself while backtesting a trading system vary in degree of subtlety. A crude one is to develop a set of trading rules by observing the past history of the market, and apply the algorithm on paper to the same past history. A more subtle one has to do with the choice of adjustable parameters. It is common to abstract the algorithm, and split it into the "artificial intelligence" (AI) core, capable of "learning", and adjustable parameters which may control the learning process, the application of its effects, or both. A developer would then prevent the AI from learning the future, and run a trading simulation program on historical data, having the AI algorithm fixed, for a range of parameter values and then pick the value that gives the best result. By not letting the AI learn the future, the developer reduces the level of selfdelusion somewhat. The best simulated performance would still incorporate "the benefit of hindsight" albeit in a refined way  through the choice of the adjustable parameters. While the AI did not learn from the future directly, its way of learning would incorporate the benefit of hindsight, and the results might contain a survivorship bias of sorts. The result? In case of a poor AI, the result will be a trading strategy with a hidden "peso problem". Below I demonstrate the quality of the Forex Automaton™ AI directly, by removing the selection step and thus the survivorship bias it brings in. I look simultaneously at the entire range of possible "ways of learning", refusing to reject those inside the range. Nor do I reject any of the possible money management styles, for the same reasons.
Each point in Fig.1 represents a simulated trading history. Points differ by the input parameters which enter the decision making algorithm. The parameters affect the style of forecasting as well as money management (stop loss, sensitivity to entrance signal, and sensitivity to exit signal). There is currently only one parameter responsible for the forecasting as such, and its nature will not be disclosed. In the spirit of the "no benefit of hindsight" testing, this parameter was varied with no restriction, including the values deemed to be reasonable and those which can be ruled out a priori or on the basis of previous experience. The decisions are made daily, 9am New York time. All incarnations of the algorithm trade with 1:100 leverage and have the same risk restriction  not more than 1/10 of trading capital in the game at any point in time. Red points are the real AUD/JPY, black points  four simulated random markets with volatility and overall trend of AUD/JPY but otherwise unpredictable by construction. They would look realistic to many traders' eyes though. Vertical axis is annual return, 1=100% a year. Horizontal axis is RMS of the return. Exactly the same algorithm trades real and Monte Carlo markets, going through exactly the same selection of over 13 thousand combinations of parameters. Real data have real GainCapital spread for the day, the simulated data  fixed spread of 0.0379. The fact that larger return requires larger risk is familiar to every investor, is seen from Fig.1 and does not need much commenting. The key theme of Fig.1 is the comparison between real and "fake" markets. It is the ability to exploit the nonrandomness of the real market that constitutes the result of the learning process and it's the difference between the red and black points collectively that demonstrates the quality of our work. Indeed, relatively few red points lie in the undesirable area of negative return, despite the latter being the predominant outcome of trading a random market. For a given degree of risk, the red points provide consistently better return. Fig.1 shows that you can be lucky and win against the random market, as is seen from many "successful" black points. Efficient market theorists would use this property of the game to reconcile the trading success of the few outstanding individuals with the "you can't beat the market" assumption  to them, an outstanding trader would represent just such a black point outlier outcome. While the argument itself is unassailable in every individual case, Fig.1 disagrees with the efficient market hypothesis for the statistical sample of simulated trading histories, for AUD/JPY on the day scale, by demonstrating the ability of an algorithm (no matter how designed) to see the difference between the real and unpredictable market, and to beat the unpredictable market of the EMT with consistently higher returns. DetailsThe time series used cover the time range of August 20, 2002 through September 01, 2008, and have the time step of one day. The actual trading begins only on March 15, 2006, after the initial learning is completed. As advertised, the algorithm has no access to the future while learning. The learning continues as the trading progresses. Fig.2 shows the random simulation charts used for the reference in the return vs risk study. On average, the simulated AUD/JPY rate grows with time because so does the real AUD/JPY on average for the period. You can see risky drops of the Aussie, which resemble reality. Chartists would identify channels, lines of resistance and support, and perhaps even Elliott waves  all totally fake.
The advantage of trading real AUD/JPY over random markets is obvious from Fig.3. The real distribution of returns is confidently located in the positive return area, with the impressively fat tail extending into the area of truly exceptional results of 1000% and even 2000%  and these are averages over two and a half years of trading! The probability of making a 100% return is twice as low with the real market as it is in the simulation. AUD/JPY was chosen for this article purely because of its alphabetical position. Combining multiple popular pairs, one can construct a portfolio with a Sharpe ratio much better than that of any individual pair, if the correlations are properly managed. Selecting good forecasting parameters and slicing out a set of trading system products that suit various risk tolerances looks like a straighforward continuation. 

Last Updated ( Friday, 11 September 2009 16:28 ) 