Minimizing the "benefit of hindsight" in trading system testing

User Rating: / 0
Written by Forex Automaton   
Monday, 27 October 2008 16:07

Once you know something, such as the past history, pretending that you do not is very difficult. The ways a trader can delude him-/herself while backtesting a trading system vary in degree of subtlety. A crude one is to develop a set of trading rules by observing the past history of the market, and apply the algorithm on paper to the same past history. A more subtle one has to do with the choice of adjustable parameters. It is common to abstract the algorithm, and split it into the "artificial intelligence" (AI) core, capable of "learning", and adjustable parameters which may control the learning process, the application of its effects, or both. A developer would then prevent the AI from learning the future, and run a trading simulation program on historical data, having the AI algorithm fixed, for a range of parameter values and then pick the value that gives the best result. By not letting the AI learn the future, the developer reduces the level of self-delusion somewhat. The best simulated performance would still incorporate "the benefit of hindsight" albeit in a refined way -- through the choice of the adjustable parameters. While the AI did not learn from the future directly, its way of learning would incorporate the benefit of hindsight, and the results might contain a survivorship bias of sorts. The result? In case of a poor AI, the result will be a trading strategy with a hidden "peso problem".

Below I demonstrate the quality of the Forex Automaton™ AI directly, by removing the selection step and thus the survivorship bias it brings in. I look simultaneously at the entire range of possible "ways of learning", refusing to reject  those inside the range. Nor do I reject any of the possible money management styles, for the same reasons.

Return vs risk for trading AUD/JPY algorithmically with real and simulated data

Fig.1:Return vs risk for paper trading in real AUD/JPY and simulated reference random walk markets. Vertical axis: annualized return, calculated as an average of independent monthly statistics, 1=100% per annum. Horizontal axis: standard deviation (RMS) of such annualized return from the same monthly statistics, same unit. Each point represents an instance of a trading system, a simulated trader; points differ by trading strategy which is subject to optimization. Black points: trading 4 fantasy markets with volatility of AUD/JPY, random by construction (no predictability or patterns). Red points: real AUD/JPY. Not all outcomes are included: in some cases, the trading did not last even a month therefore no RMS could be calculated -- there would be no other month of trading performance to compare with. In such cases the system is programmed to record a zero RMS, and such outcomes did not make it into the figure. The only other requirement is that of excluding the mean annualized returns above 5 (500%) and RMS above 20 -- happy as I am to include higher returns, they would make the picture difficult to analyze visually.

Each point in Fig.1 represents a simulated trading history. Points differ by the input parameters which enter the decision making algorithm. The parameters affect the style of forecasting as well as money management (stop loss, sensitivity to entrance signal, and sensitivity to exit signal). There is currently only one parameter responsible for the forecasting as such, and its nature will not be disclosed. In the spirit of the "no benefit of hindsight" testing, this parameter was varied with no restriction, including the values deemed to be reasonable and those which can be ruled out a priori or on the basis of previous experience. The decisions are made daily, 9am New York time. All incarnations of the algorithm trade with 1:100 leverage and have the same risk restriction -- not more than 1/10 of trading capital in the game at any point in time. Red points are the real AUD/JPY, black points -- four simulated random markets with volatility and overall trend of AUD/JPY but otherwise unpredictable by construction. They would look realistic to many traders' eyes though. Vertical axis is annual return, 1=100% a year. Horizontal axis is RMS of the return. Exactly the same algorithm trades real and Monte Carlo markets, going through exactly the same selection of over 13 thousand combinations of parameters. Real data have real GainCapital spread for the day, the simulated data -- fixed spread of 0.0379.

The fact that larger return requires larger risk is familiar to every investor, is seen from Fig.1 and does not need much commenting. The key theme of Fig.1 is the comparison between real and "fake" markets. It is the ability to exploit the non-randomness of the real market that constitutes the result of the learning process and it's the difference between the red and black points collectively that demonstrates the quality of our work. Indeed, relatively few red points lie in the undesirable area of negative return, despite the latter being the predominant outcome of trading a random market. For a given degree of risk, the red points provide consistently better return.

Fig.1 shows that you can be lucky and win against the random market, as is seen from many "successful" black points. Efficient market theorists would use this property of the game to reconcile the trading success of the few outstanding individuals with the "you can't beat the market" assumption -- to them, an outstanding trader would represent just such a black point outlier outcome. While the argument itself is unassailable in every  individual case, Fig.1 disagrees with the efficient market hypothesis for the statistical sample of simulated trading histories, for AUD/JPY on the day scale, by demonstrating the ability of an algorithm (no matter how designed) to see the difference between the real and unpredictable market, and to beat the unpredictable market of the EMT with consistently higher returns.


The time series used cover the time range of August 20, 2002 through September 01, 2008, and have the time step of one day. The actual trading begins only on March 15, 2006, after the initial learning is completed. As advertised, the algorithm has no access to the future while learning. The learning continues as the trading progresses. Fig.2 shows the random simulation charts used for the reference in the return vs risk study. On average, the simulated AUD/JPY rate grows with time because so does the real AUD/JPY on average for the period. You can see risky drops of the Aussie, which resemble reality. Chartists would identify channels, lines of resistance and support, and perhaps even Elliott waves -- all totally fake.

Simulated AUD/JPY, case 1 Simulated AUD/JPY, case 2 Simulated AUD/JPY, case 3 Simulated AUD/JPY, case 4

Fig.2:Simulated martingale (random walk) markets with volatility of real AUD/JPY used in the study. If you can tell how these simulated histories differ from the real AUD/JPY and this is not just because you remember what the real AUD/JPY looks like, you may not need our trading signal.

Annualized return distribution for real and martingale market

Fig.3:Comparing annualized return distributions for real AUD/JPY and simulated martingale markets. The histograms are normalized to the same (unity) total integral. The cut-off at 20 is intentional and omits the most exceptionally successful outcomes to keep the figure legible. Unit is 100% per annum.

The advantage of trading real AUD/JPY over random markets is obvious from Fig.3. The real distribution of returns is confidently located in the positive return area, with the impressively fat tail extending into the area of truly exceptional results of 1000% and even 2000% -- and these are averages over two and a half years of trading! The probability of making a -100% return is twice as low with the real market as it is in the simulation. AUD/JPY was chosen for this article purely because of its alphabetical position. Combining multiple popular pairs, one can construct a portfolio with a Sharpe ratio much better than that of any individual pair, if the correlations are properly managed. Selecting good forecasting parameters and slicing out a set of trading system products that suit various risk tolerances looks like a straighforward continuation.

Bookmark with:    Digg    reddit    Facebook    StumbleUpon    Newsvine
Last Updated ( Friday, 11 September 2009 16:28 )