Temporal (in)stability of trading system optimization curves

User Rating: / 1
PoorBest 
Written by Forex Automaton   
Friday, 21 May 2010 09:52

For the first time, I address the question of how stable the optimization results are in time. While predictabilities of daily high and low show a highly stable pattern of dependence on the parameter subject to optimization, the positive results for close are mainly due to the high impact of a single period, which happens to cover the financial panic of the last quarter of 2008.

This study relies on the version of the forecasting algorithm not yet deployed on the production host. The simulation approach is the usual one, with the same 14 currency pairs tracked by Danica and daily low-high-close data set covering the interval from August 20, 2002 through May 17, 2010. To study the time stability of optimization, the interval has been split into five equal pieces. The simulated trading period begins after the initial training period is over in middle of April 2006, and this period is split into 5 sub-periods of equal length. Sub-period 4 for example begins in October 2008 and lasts for somewhat less than ten months.

The figure of merit used, Pearson correlation coefficient between predicted and actual logarithmic returns of daily low, high and close seems to need no further introduction -- there is hardly anything new to be said about them in this article.

Correlation of predicted and actual logarithmic returns in daily high, as a function of trading system parameter Fred -- all trading days 1.1 Correlation of predicted and actual logarithmic returns in daily close, as a function of trading system parameter Fred -- all trading days 1.2

Fig.1. Different sets of symbols denote different sub-intervals, numbered from 1 to 5, of the total trading time interval from April 15, 2006 to May 17, 2010. Panel 1.1 shows dependence of the Pearson correlation coefficient between predicted and actual logarithmic returns in daily high on the optimization parameter nicknamed Fred, responsible for the forecasting, while panel 1.2 shows the same for daily close. Arrows indicate the choice of Fred implemented in the current version of Danica and affecting the forecasts we publish daily.

Correlation of predicted and actual logarithmic returns in daily high, as a function of trading system parameter Fred -- trading days with L0 trigger word=3 2.1 Correlation of predicted and actual logarithmic returns in daily close, as a function of trading system parameter Fred -- trading days with L0 trigger word=3 2.2

Fig.2. Different sets of symbols denote different sub-intervals, numbered from 1 to 5, of the total trading time interval from April 15, 2006 to May 17, 2010. Just like in Fig.1, top panel shows dependence of the Pearson correlation coefficient between predicted and actual logarithmic returns in daily high on the optimization parameter nicknamed Fred, responsible for the forecasting, while the bottom panel shows the same for daily close. The difference with Fig.1 is that now L0 trigger word 3 is required in order for the "event" (day) to enter the plots. Selected data represent about 80% of all data.

Sub-period 4 is the period that includes the financial panic of 2008; it is this period that looks most lucrative in simulations. There is no guarantee that the values of Fred that maximized returns during the panic will continue to work. In particular, the subsequent "recovery" period (sub-period 5 in figures) is seen to have its maximum Pearson at a different (lower) Fred value. These data show no single safe Fred zone for close: any choice of Fred would lead to positive results for two of the five periods and zero or even somewhat negative results for three other ones.

This disturbing lack of a single optimization curve shape for daily close is in sharp contrast with the highly stable character of the similar curves for daily extremes (only daily high is shown in the figures). Not only are those curves stable, they also show much higher degree of correlation between predicted and actual logarithmic returns.

A daily range strategy may be based on placing a market order with a stop-loss set at the previous day's extreme predicted to not be reached during the day and a profit-target set at the previous day's extreme predicted to be hit. Nominally this avoids the issue of the forecast quality for close. However, a comparison of Fig.1.2 and Fig.2.2 (and similar results from an earlier study) indicates that predictability of extremes is improved by requiring the forecast for close to point in the same direction as high and low. If that path is taken, then the question becomes -- how is it better than betting on close?

The Day Range strategy with no selection and re-examined this week, was seen to result in a large proportion of wins and if one calculates Kelly allocation for it, it comes out in excess of 0.3 for some currency pairs when trading in the direction of "risk appetite" (long commodity currencies and interest differential), and in the ballpark of 0.15-0.25 when trading in the direction of "risk aversion". (This peculiar dependence of Kelly coefficient on the trading theme of the day has been already touched upon and awaits deeper investigations.) The problem of the Kelly Criterion in this context however is that the trades with two limits apparently break the cornerstone assumption of Kelly approach, namely the one of the parimutuel distribution of profits and losses (see Eq.1 of the article on Kelly Criterion). As a result, the high Kelly coefficients correspond to trading strategies with net loss: broker's spreads and the infrequent losses end up doing so much damage that it is not fully offset by the much more frequent profits. Once the profit target is dropped, Kelly begins to make more sense as the profit distribution acquires a tail which, even though does not completely turn it into Kelly's "fair odds" (minus first power) distribution, at least seems to put it within the same broad class of animals.

In conclusion, I list some ways of making future progress.

  • What kills the Day Range approach (the more sophisticated version with trade idea selection) is broker's spreads. The spreads have been falling recently; if they continue to do so without impairing the high/low predictability effect shown in the figures, the Day Range strategy, with capital allocation properly adjusted to predictability, will one day become profitable for some or most of the forex pairs. One needs to look closer at the picture of spreads and perhaps drop some of the high spread currency pairs from consideration.

  • Leaving the optimization aside, there are "brute force" ways of improving prediction quality -- they essentially consist in increasing the size of the objects the prediction algorithm analyzes. While we are clearly in the zone of diminishing returns along this path, one may be able to squeeze out  a few percent improvement in the quantities plotted above. Essentially this will raise the entire curve a few percent at the cost of more CPU consumption in each act of prediction.

  • Kelly capital allocation coefficients have not been used in this study. It remains to be seen whether or not some application of Kelly coefficients will make the shapes more uniform. A curious detail is the observation that risk-averse environments in general are discouraged by this system's Kelly capital allocations, as already mentioned. Therefore if Kelly's coefficients are positive, that's likely to be not just because of the financial panic that took place in 2008. Will Kelly coefficients balance the performance between the periods? That remains to be seen.

  • Excercises with calculating Kelly coefficients for the Day Range strategy mentioned above highlighted an inherent problem with Kelly Criterion, and became the first practical situation where the approach looked woefully inadequate. Instead of the analytic Kelly approach with a simple formula and questionable assumptions, the preferred solution may look like a computerized maximization of terminal wealth relying entirely on real-life input -- a prototype of such a procedure has been demonstrated already.

Bookmark with:

Deli.cio.us    Digg    reddit    Facebook    StumbleUpon    Newsvine
Last Updated ( Saturday, 21 January 2012 16:55 )