ForexAutomaton 2013. The Fifth Annual Summary of Researh Progress

User Rating: / 3
Written by Forex Automaton   
Sunday, 14 April 2013 16:53

Success of algorithmic trading has as much to do with the market as with the nature of the algorithm. While the algorithms may vary and may have terms such as artificial intelligence, adaptivity, machine learning, neural networks and the like associated with them, the phenomenology of what it is that the machines are "learning" all comes down to correlation measurements of varying order -- pairs, triplets, quadruplets... -- over a variety of time lags and time scales, performed on the market time series.

ForexAutomaton began its life on April 02, 2008, five years ago. Year V was exceptionally productive for the project. Here are the main highlights of this annual review:

  • An Index of Correlation Strength (CERCSI) and a Predictability Index have been formulated and the history of these measures has been established.
  • Index of Correlation Strength reveals that a regime switch in the market dynamics took place in Summer 2007.
  • Danica, the daily forecast system, has confirmed its status of an "Anti-System", delivering its forecasts in the non-random and consistent opposition to the market reality.
  • The challenge of regime switch has been addressed by implementing an algorithm, nicknamed Twisted Pair, to incorporate sensitivity to two basic types of memory effects: trending and reversion to the mean.
  • Danica, the daily forecast system, and Heidi, the hourly system, have been upgraded with the Twisted Pair algorithm.

FX Market Diagnostics: In The Wake Of The Year 2007 Regime Switch

CERCSI or the Currency Exchange Rate Correlation Strength Index is one of the quantities monitored monthly in the Forex Correlations section of the site. CERCSI measures the absolute magnitude (deviation from zero in either positive or negative direction) of correlation strength on the hourly scale among the 14 leading spot FX rates. The quantity is computed monthly and includes a month of hourly data. By construction, CERCSI is a quantity bounded between 0 and 1.

CERCSI, Correlation Strength Index Logarithmic Volatility

Fig.1. Top panel: history of the Correlation Strength Index since 2003. Bottom panel: history of the logarithmic volatility for the same period. Both quantities are computed using the same 14 time series of spot foreign exchange data on the hour scale.

Fig. 1 shows that a change in the correlation regime with a dramatic increase in the absolute strength of inter-market correlations occured in Summer of 2007 and apparently was triggered by the subprime mortgage crisis in the US. CERCSI signaled an onset of the new regime months before the volatility spike of 2008. From the CERCSI point of view, we are still more or less in the same regime, despite the fact that the volatility has returned to normal.

As a reminder, CERCSI (just like CERPI and volatility) is insensitive to the absolute level of prices and works only with logarithmic differentials thereof. A high absolute magnitude of correlations indicates a degeneration in the variety of market instruments, or rather, of their perception by the market participants. Everything degenerates into "safe haven" and "risk assets", and effectiveness of portfolio diversification is thus reduced. And when diversification stops working, portfolio managers realize that they can not maintain prior risk profiles of their portfolios without selling risk assets or shorting them. Which is why high CERCSI may signal trouble for investors as it did in 2007.

CERPI, Correlation Predictability Index

Fig.2. History of the Correlation Predictability Index (CERPI) since 2003. The dotted line is the estimated level of CERPI consistent with efficient market hypothesis.

The fact that algorithmic trading does or does not work may have to do with the market as well as with the nature of the algorithm. While the algorithms may vary and may have terms such as artificial intelligence, adaptivity, machine learning, neural networks and the like associated with them, the phenomenology of what it is that the machines are "learning", and whether there is anything to learn, all comes down to correlation measurements of varying order (pairs, triplets, quadruplets and so on), over a variety of time lags, and over a variety of time scales. The zero benchmark of the hypothetical efficient market corresponds to the absence of non-trivial correlation effects at non-zero time lags. Our monthly Forex Correlation reviews are currently limited to second order, or pair-wise correlation effects, and include auto-correlations and inter-market correlations with non-zero time lags. The simplest benchmark of market predictability is the one based on such correlations, and it is not difficult to construct. CERPI is just such a quantity, and its chart is shown in Fig.2.

CERPI, computed on a given time scale, is not sensitive to all types of predictability effects on all time scales, but nevertheless it provides a useful metric. Fig.2 shows that after the Regime Switch in Summer 2007, the FX markets do occasionally deliver months with high values of CERPI, although perhaps in a less reliable manner than before: the positive extremes reach higher but they are also less frequent.

After this general quantitative characterization of the conditions we are in, we continue with the brief reviews of all three trading systems currently deployed: Heidi, Demi and Danica. We will conclude with the Twisted Pair idea so recently implemented for Heidi and Danica.


Heidi is the prediction-only system operating on the hourly time scale publicly since August 11, 2010. In May of 2011, the system was upgraded to version 2, but the upgrade did not affect the forecasting concepts or parameters. A parameter upgrade was made on January 21, 2012. This affected just one of the seasonality windows, window 6. After that, the performance in window 6 was seen to have improved.

Heidi forex trading system, figure of merit: degree of correlation between the predicted and real logarithmic price difference at the close of the hour, for the 8 intra-day time windows

Fig.3. Heidi's figure of merit: degree of correlation (Pearson correlation coefficient) between the predicted and real logarithmic price differences at the close of the hour, for the eight intra-day time windows. This figure is from December 2012 review of the system.

Central Europe 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 0
Eastern US 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
intra-day season ID 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 0 0

Table 1. Definition of intra-day seasons and time zone conversion table. Seasonal time shifts, such as daylight saving time, may complicate the picture if the nations choose to enact them on different days, and are ignored.

Overall, version 2 of Heidi looks like a realistically successful system, leaving aside the question of whether these positive Pearson correlations of forecast with reality can allow the user to beat the bid-ask spread given the time frame of one hour on which the system operates. But behind this success lies a rather crude implementation of the mean-reversion model: take the "raw" forecast and flip the sign. The Twisted Pair mechanism, to be described, provides a much more sophisticated alternative. Version 3 of Heidi, with Twisted Pair, was launched on April 12, 2013.


Demi's origin and design has been discussed in great detail in one of the earlier Annual Reports. In brief, the idea behind this system is this: knowing that our prediction for a change in daily high is correlated with the actual change, and knowing that the prediction for a change in daily low is correlated with the actual change, we bet on a coincidence of the successful predictions for these two changes. Note that the existence of such a coincidence is, strictly speaking, a separate matter from the individual successes of both types of predictions. What makes this better than gambling is positiveness of the fourth-order cumulant among the forecasts and real changes in both high and low. We bet on the coincidence by opening a trade with a stop-loss and a profit target. The stop loss and the profit target are the previous day's high and low, and the direction of the trade is long if today's low and high are expected to be above yesterday's; short if they are expected to be below yesterday's. Note the requirement of simultaneity: in a situation when high and low are expected to evolve in the opposite directions, a trade is not submitted.

The four instances of Demi, differing by the hour of the decision making (24-hour day definition) systems went live on August 25, 2010. In the following, the systems will be distinguished according to the hour of decision making during the day (the end and beginning of the day): 3, 5, 9 and 11 (all in the AM). System performance data presented below describe the situation as of March 30, 2013. This should be compared with the the previous Annual Report.

Trading costs are simulated as bid-ask spread, with the spread being equal to a fixed fraction of price (see link for details about the simulation of the spread).

Table 2. Demi's historical performance as of March 30, 2013.

ID Capital (was 1 at the inception)Profit/loss in quote units, sum total since inception Profit/loss in quote units, average per trade 90-day return, per annum, % 180-day return, per annum, % 360-day return, per annum, % # trades # trades since going live
3 1.346 0.0866 2.07 × 10-4 -12.2 -14.6 -5.94 418 139
5 1.15 0.114 2.59 × 10-4 0.724 -5.14 -4.70 440 177
9 0.932 -0.0267 -6.12 × 10-5 -14.8 -13.3 -1.31 436 225
11 1.355 0.0245 6.53 × 10-5 -7.16 -2.36 -1.90 375 142

Table 3. Demi's performance progress since the last annual report.

dateTotal capital (was 4 at the inception)Total profit/loss in quote units, sum total since inception
March 30, 2013 (this report) 4.78 0.198
April 02, 2012 (second report) 5.07 0.225
March 25, 2011 (first report) 4.88 0.205

Table 3 gives a birds-eye picture of Demi's progress for the year since the last report. This is a sum total of capital under management and accumulated P&L in quote units over the four systms. Both statistics reflect trading costs, but the P&L in quote units is a much simpler statistic, not reflecting capital allocation decisions (leverage) which differ trade by trade. Demi has suffered a 5.7% loss this year.

Danica. The Anti-System.

The Danica model has been making daily forecasts on this site for three years. I said it before and will say it again... When launching the model, I expected one of two outcomes: either the model would prove successful by generating a statistically significant, even if "small", positive correlation with reality, in which case it would be declared a success -- or it would look statistically uncorrelated with reality, in which case it would be no better than a random number generator, with no merit as a forecasting tool.

The reality proved to be quite different from both of these scenarios.

Danica forex trading system. History of correlation between forecast and reality.

Fig.4. Correlation coefficient between the predicted and actual logarithmic difference in daily close, month by month, from 2010 through early 2013, for the 14 exchange rates tracked by the Danica system. The bias of the system to produce forecasts which "oppose" the subsequent reality is seen graphically as the tendency of the bars to point downward.

In Fig.4, the correlation coefficient is the quantity given in the Danica monthly reports, and is, by our adapted standard, the Pearson correlation coefficient over the last 24 business days of the model's operation, as reported in the last forecast of the month. The important thing is that the values of such coefficient for the different months (the different bars in Fig.3) are statistically independent.

The actual outcome of over three years of live operation of the system is neither of the two naively expected scenarios: the model is neither systematically right nor worthless. The model forecasts have been systematically opposite to what happens later in reality.

The data are not consistent with the academic notion of an hypothetic efficient market, understood as a guaranteed source of mathematically random, symmetric gains and losses, irrespective of the trading strategy.

In the last year's report, I wrote: "The FX market, from Danica's point of view, seems to be a willful entity with meta-stable patterns of behaviour which are replaced by other meta-stable patterns. The ability of the system to convert input (FX quote) data with no visible two-point correlation on the day scale (speaking about the direction of closing price), into a stream of output data (trading gain or loss) with visible positive auto-correlation (regadless of the sign of gain or loss itself) is an important achievement, as the problem becomes that of coming up with a transformation to change the broadly autocorrelated, even though stochastic and bipolar, quantity, into a positive one. One could think of some afterburner to Danica which would decide whether to treat Danica as a "system" or as an "antisystem" at this point in time."

The "afterburner" was indeed the kind of solution adapted for Heidi 2, and its relative success has been already discussed in the context of that model. For Heidi 3 and Danica 2, a different, much deeper and more elegant solution, nicknamed Twisted Pair, has been implemented.

Twisted Pair

Twisted Pair is a particular transformation applied to the input data. Naturally, a corresponding transformation is applied to the output data of the algorithm. The proof of principle for the Twisted Pair has been obtained with simulated data and is illustrated in Fig.5.


Pearson correlation for close, simulated trending time series Pearson correlation for close, simulated mean-reverting time series

Fig.5. Proof of principle for the Twisted Pair. Tuning the system on Monte Carlo data with a single parameter, Fred. The figure of merit is the Pearson correlation coefficient between real and predicted logarithmic return for an elementary period in the time series. Top: trending data sample. Bottom: mean-reverting data sample.

By mean reversion I will mean the tendency of the price time series to form quickly alternating rises and falls, more pronounced than in a fully unpredictable time series of the same volatility. In an intuitive sense, such a time series looks as if a rise during this hour is caused by the previous hour's fall, or this hour's fall is caused by the previous hour's rise, since rises and falls tend to cluster in an alternating fashion. Mathematically, using correlations, a manifestation of this behaviour would be an auto-correlation of (logarithmic) returns looking like a dumped oscillation.

Mean-reverting markets can be traded algorithmically or intuitively once the characteristic time scale (period of reversion) has been reconstructed or otherwise recognized. On an intuitive level, the trading involves the psychological attitude known as "being a contrarian". Pair trading involves artificial market instruments (pairs) whose mean-reversion properties are enhanced, compared to those of the ingredients.

Trending markets are the opposite extreme and present a situation psychologically comfortable to many people, where you can bet on the continuation of a trend, once the trend has been recognized. Mathematically, trends look like broad positive correlations of (logarithmic) returns.

The simulated time series and the method used for synthesizing them have been discussed in the ForexAutomaton report of October 28, 2012.

For the ForexAutomaton systems, the challenge has been to recognize the mean-reversion pattern "natively". Twisted Pair was the first algorithm that solved this problem, as seen in Fig.5. The system has no prior knowledge of what kind of time series would be fed into it. Yet, in both cases, positive correlation of forecasts with reality is obtained. Curiously, the optimization curves in Fig.5 look like mirror reflections of one another. Possibly, this is a reflection of the deep inner opposition and thus complimentarity of the two basic patterns, that of trend and that of mean-reversion.

How adequate is this solution? How to take full advantage of its potential? The new incarnations of Danica and Heidi will attempt to provide material for answering these questions.

Web Site Access

At present, Danica is in public access and accessing Demi and Heidi requires a free registration. I intend to keep providing these services free for another year. Should that change, pre-registered users will be granted a free year of access from the moment the free access is terminated.

Bookmark with:    Digg    reddit    Facebook    StumbleUpon    Newsvine
Last Updated ( Friday, 12 June 2015 15:51 )