Heidi performance review

User Rating: / 2
PoorBest 
Written by Forex Automaton   
Saturday, 21 January 2012 17:00

Data accumulation for the present incarnation of Heidi began on December 1, 2010 and the first predictions (in back-testing mode) were generated for March 16, 2011. The present system was announced and began live operation on May 23. Now, eight months later, I take a look at the first statistically significant performance figures of merit and make an adjustment to one of the system parameters.

 

As has become customary in this project, I use Pearson correlation coefficient between predicted and real changes in FX quote under study as the main figure of merit on the stage of forecasting optimization. At this point in time, with less than 3 month of back-testing and 8 months of real-life operation, the total statistics is dominated by real life.

Correlation coefficients between predicted and real values of the logarithmic difference of hourly close with respect to the previous hour (logarithmic return), denoted as C(forecast,reality|close), are plotted in Fig.1 for the 8 time windows, numbered from 0 to 7. The time windows are defined in the output manual of the system. The correlation coefficients plotted are averaged over the 14 currency pairs tracked by the system.

 

Correlation coefficients between predicted and real values of the logarithmic difference of hourly close with respect to the previous hour (logarithmic return), vs the intraday time window

Fig.1. Heidi's figure of merit: degree of correlation between the predicted and real logarithmic price differences at the close of the hour.

Time windows 2, 6 and 7 have statistically significant deviations of correlation coefficients from zero. This is inconsistent with market efficiency postulate which would make you expect that performance of any kind of market timing system converges to zero with time. Out of these "inefficient" windows, window 2 and window 7 have significantly positive C(forecast,reality|close), indicating successful forecasting. The overall slightly negative performance of the model is dominated by the contribution of the single time window, time window 6. 

There are either hints of deviation or significant deviations  from zero for the time windows 1, 2, 3, (2:00 to 10:00 Central European time  by the close of hour) and 6, 7 (17:00 to 22:00 Central European time). These times correspond to the morning hours in Japan and Central Europe, and morning hours in the US. There are no hints of such inefficiencies  for windows 0, 4, and 5.

Intra-day time window 6 has been given a special treatment in the Heidi optimization. It was decided that this is the only time window during the day when the concept of trend following makes sense on the hour-by-hour time scale. Therefore, this window was given a very special treatment, allowing the system to play trend following instead of trend-reversion. Technically, the difference amounts to whether the sign of the "raw" forecast generated by the system has its sign flipped. Acting under the assumption that meta-stable features in the markets may live much longer than it takes to detect them, I decide to disable the special treatment of window 6, and watch the difference it makes on the system performance. Obviously, it will only have an effect on window 6 which in a sense could not be worse; the effect should be to flip the correlation coefficient up above zero.

Bookmark with:

Deli.cio.us    Digg    reddit    Facebook    StumbleUpon    Newsvine