How I know my forex forecasting engine works

User Rating: / 0
PoorBest 
Written by Forex Automaton   
Tuesday, 20 October 2009 11:09

The key component of my forex trading system is the prediction engine. So far I have been trying to optimize the prediction engine in the overall context of simulated trading on the historical data where besides the parameter responsible for prediction, at least three other parameters were being optimized. With the large number of parameters, the CPU demands of optimization become prohibitive: as the dimensionalty of the parameter space grows, the number of parameter combination grows with it. As the pseudo-random time series are at the very heart of the problem, the randomness of changes in performance with respect to every parameter clouds the analyst's vision of any parameter in the course of the optimization. The adverse effect of that is possibly even more important than the combinatorial growth of the volume of the parameter space. It is therefore very helpful to factor the problem out into independent pieces which can then be optimized separately. The success depends, among other things, on the figures of merit used and on the degree of true independence between such pieces.

Predictions for every market under analysis are obtained at every decision-making step (in this case, a day). As always, the system has no access to the future of the time series and only learns from the past. Every step during the simulation is therefore a direct test of the applicability of the past learning to the present context, just as it will be in real life. In real life however, the system chosen for operation will bear in itself the bias associated with its selection. In the Monte Carlo tests, we don't select and deal with an entire array of systems. The statements made for such an a priori array are free of selection bias.

In order to conduct an unbiased test of the prediction quality and determine the best prediction parameter, I use the following procedure. At every prediction step, I record the prediction and one step later, when the future becomes reality as predicted or otherwise, I take the product of the real and predicted logarithmic returns for the day. The average of the product is the quantity plotted along the vertical axis in Fig.1 for the entire range of the prediction parameters.

Covariance of predicted and actual day-scale logarithmic returns as a function of  the forecasting parameter

Fig.1. Covariance of predicted and actual day-scale logarithmic returns as a function of the forecasting parameter nicknamed Fred. Back-testing simulations with no access to the future.

The product of two numbers with the same sign, either negative or positive, is positive. The product of two numbers with the opposite signs is negative. The numbers are logarithmic retuns, therefore a positive sign corresponds to the upside trend, the negative sign -- to the downside trend. Predicting the sign is like predicting the trend. But the magnitude also matters: when both the prediction and the reality have high magnitude, the product is also large. The goal is therefore to have a large positive product in as many cases as possible. The average of such a product, especially for time series with negligible mean, is often called covariance:

Cov(prediction,reality) = E(prediction√óreality),

where E() denotes the operation of expectation -- in this case, averaging.

Every day for every currency pair is an entry in a histogram like Fig.2, and Fig.1 shows the averages of those entries for every Fred value. A point in Fig.1 is therefore an average (with equal weight) of the six means in each of the histograms like those in Fig.2, except that in Fig.2 I am showing only the histograms for Fred=60 which is seen in Fig.1 to be the best.

Now to the analysis of these data. The first feature of Fig.1 is the fact that the majority of points, 12 out of 16, lie above zero indicating quality forecasting. Not all Fred parameters are equally good, with the one at 60 being the best. All six currencies are used in Fig.1 even though, as you see in Fig.2, dropping some of them could "improve" the results -- but I see no a priori reason to drop them and want to keep the analysis totally bias-free, therefore nothing is dropped.

The vertical bars in the figure denote the estimate precision of the knowlege of the mean on the basis of six measurements, assuming the measurements, coming from independent decision making on the somewhat correlated currency pairs are independent. The uncertainty of a point is with some accuracy its vertical bar, the so called errorbar. I leave the question of how Gaussian the distribution of an individual point is outside the scope of this post.

The visual impression of someone trained in looking at scientific data with one sigma errorbars is that the individual points do not vary enough to justify the large errorbars. This is of course true, and the reason is lack of independence between the differend Fred values -- all of them are obtained using the same time series. So a physicist would consider that a systematic component of the error common to all points. Without going too deep into details, and especially avoiding any discussion of the nature of Fred, I can say that certain points very far away from the optimum can be used as an estimate of the systematic error (you need to pick points for which there would be no justifiable reason to deviate from zero other than pure luck, then a good fraction of that "luck" is systematic error). The systematic error seems likely to be negative and fairly large, thus there are reasons to belive that the quality of prediction is even somewhat better than it looks.

A final observation is that this result is not too different from the one obtained on the basis of full back-testing simulation of the trading system performance.

To summarize, this is a solid positive result which is, as everything in this business, is subject to statistical and systematic uncertainty. Unlike many previous results, this exciting result is obtained in a way which is unaffected by the details of the trading simulation and by the money management strategies which may be in need of independent optimization. My subsequent strategy is therefore to stick to guns optimizing the money management.

The historical forex data used in this report covers the time interval from late August 2002 to late August 2009, and the decision making by the trained system begins in April 2006.

Distribution of a product of predicted and actual day-scale logarithmic returns for AUD/USD at a fixed forecasting parameter 2.1 Distribution of a product of predicted and actual day-scale logarithmic returns for EUR/USD at a fixed  forecasting parameter 2.2 Distribution of a product of predicted and actual day-scale logarithmic returns for GBP/USD at a fixed  forecasting parameter 2.3 Distribution of a product of predicted and actual day-scale logarithmic returns for USD/CAD at a fixed  forecasting parameter 2.4 Distribution of a product of predicted and actual day-scale logarithmic returns for USD/CHF at a fixed  forecasting parameter 2.5 Distribution of a product of predicted and actual day-scale logarithmic returns for USD/JPY at a fixed  forecasting parameter 2.6

 

Fig.2. Distributions of the product of real and predicted logarithmic returns on the day scale for Fred=60. 2.1:AUD/USD, 2.2:EUR/USD, 2.3:GBP/USD, 2.4:USD/CAD, 2.5:USD/CHF, 2.1:USD/JPY. The tails look long even on the logarithmic scale.

Bookmark with:

Deli.cio.us    Digg    reddit    Facebook    StumbleUpon    Newsvine
Last Updated ( Wednesday, 20 January 2010 18:24 )