Forex Automaton

While it is true that past performance does not indicate the future, the only reliable information we have is about the past. A few important things make a difference between unbiased trading-system testing and self-delusion. Here I summarize my current understanding.

Causality

When testing on historical data, we perform a series of simulated steps of decision-making and order placement steps. These time steps are discrete, contiguous and advancing in time. It is important that at any point in time the decision-making algorithm be not allowed to use information from the "future" that is the remainder of the time series. The model is forced to bear full consequences of its past actions, through increases or losses in the trading capital, affecting future money management.

Losses and stop-loss orders

Every time interval is characterized by the "high" and "low" levels of price. Unless stated otherwise, we assume that whenever a stop loss level for a position is found to lie between these levels, a stop-loss order is executed during the time period in question (same business day, same hour etc) at the price specified.

Input: adjustable parameters

It is well known that with enough parameters one can fit anything. With a large number of parameters responsible for decision making, one runs a danger of retrofitting the model (fitting to the past performance) at the expense of usability for the future. We distinguish parameters which set up the problem (such as the duration of the time series used in the forecasting algorithm), parameters related to the money management (such as the amount of capital at risk at any point in time, the leverage, and the stop-loss placement), and adjustable parameters responsible for the forecasting as such. We have only one (yes, one) parameter in this latter category.

The parameter vectors form a multidimentional, uniformly filled grid. A set of output values resulting from a back-testing run is associated with each node of the grid. These are the figures of merit we monitor. A MySQL data base with a custom-designed C++ API is used to analyze, interpret, and keep track of these data.

Output: figures of merit

We monitor:

• Average profit/loss per time step
• Average yearly return on investment over the number of years
• Maximum yearly return on investment over the number of years
• Minimum yearly return on investment over the number of years
• Sharpe ratio or a simpler ratio of return to its standard deviation
• and a number of others.

Analysis

The analysis consists in selecting an area of the multidimentional parameter space with favorable values of the figures of merit and projecting that onto the space of input parameters (that is, finding the area in the space of input parameters most densily populated by the favorable outcomes). Of course, there is no mathematical guarantee that both areas are compact and there is a correspondence. However, when it exists, that's a good news.

Martingale runs

Any meaningful correspondence or pattern in the space of input and output parameters found during back-testing with real data must disappear when switching the analysis from real data to martingale (random walk, or unpredictable by construction) data sets. Ideally, the system must be clever enough to tell whether it is dealing with real or martingale market -- the difficulty however lies in the fact that such a decision can only be made on the basis of a long enough observation of the market, and the conditions may change as time goes on. Ironically, most amateur developers are happy when the system is making money during some period in the past, without even posing the problem of whether that money was made due to intelligence of the system or due to sheer luck. On the contrary, the problem of discriminating against the random market is the cornerstone of our approach.

Bookmark with: