Reading Assignment: Common Backtesting Mistakes

#1

In this reading assignment we will dive deeper into the most important issue when it comes to building a profitable trading strategy - Backtesting. We will look at some of the common mistakes, over-optimization and curve-fitting and how you can avoid them. Read through this blog post and answer the following questions in this forum thread. Use the knowledge you have learned so far as well.

  1. What is so dangerous about over-optimization?
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
  3. Why should you avoid asymmetric trading signals?
#2
  1. curve fitting; it maybe over adjusted or tailored to past events (back testing) and may not be suitable for the future markets or new future data sets.

  2. snapshot 6 months; with a comparison to the longest possible (>5 years ideally, 9 to 11 years) data set humanly possible!

Focus on the development of a trading system aiming for adaptability, broad optimizations, robust profitability and large periods of testing data increasing better chance or probability of achieving a Ferrari.

  1. Asymmetric system; Asymmetric information can lead to adverse selection, incomplete markets and is a type of market failure.

New Algo Team; join the A-Team.

Serious about Algo optimisation keep in touch & say hello; opentrade@protonmail.com

1 Like
#3
  1. Over optimization can lead to “curve-fitting”, which is the unwanted tuning of the strategy to fit specific past data.

  2. According to the article, approx 10yrs on higher time frames for accuracy.

  3. Asymmetric signals provide more complexity which tends to lean towards tuning the strategy for specific datasets… curve-fitting.

Funny I haven’t heard of this term “curve-fitting” before, but I’ve certainly been victim of it while fine tuning strategies in the past. Interesting ;o)

1 Like
#4

What is so dangerous about over-optimisation?

  • There is a danger of over-optimisation is to produce a trading strategy with absolutely astonishing results that will not be achievable going forward.

How long should a testing period be if you are serious about building a profitable trading strategy?

  • Time frames should be greater than 30 minutes
  • Ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.

Why should you avoid asymmetric trading signals?

  • Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitting solutions.
1 Like
#5
  1. What is so dangerous about over-optimization? Over-optimization is dangerous because it trasforms the strategy in a curve-fitting of past data that will be never identic to the future data. So the very good results on the past data will not reply in the future.

  2. How long should a testing period be if you are serious about building a profitable trading strategy? In the traditional market a testing period of 9-11 years should be used.

  3. Why should you avoid asymmetric trading signals? Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

1 Like
#6
  1. What is so dangerous about over-optimization?
    Creating a false sense that plan will work just customized for past data.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    About 10 years

  3. Why should you avoid asymmetric trading signals?

You end up matching up with previous market cycles.

1 Like
#7
  1. What is so dangerous about over-optimization?
  • the strategy will be perfect for past data but not the future trading.
  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • ten years data
  1. Why should you avoid asymmetric trading signals?
  • It will lead to curve fitting as the asymmetric signals is work for the past data but we cannot sure about whether it works for the future.
1 Like
#8

1. What is so dangerous about over-optimisation?
It will only be optimised for the specific data set. This is not a clear indication of what will happen in the future. Your algorithm becomes more of a curve-fitting program; this will mean losses in the future as it eliminates the probabilistic nature of the market.

2. How long should a testing period be if you are serious about building a profitable trading strategy?
Ideally 9 -11 years if possible. Author also states a minimum of 5 years to avoid curve fitting: https://mechanicalforex.com/2010/06/five-common-mistakes-in-system.html

3. Why should you avoid asymmetric trading signals?
These trading signals should be avoided as they are generally based around the economic variables at that time period for the particular currency (interest rates etc). A good example is over optimisation of previous data sets. This can create two separate sets of criteria in the market for “shorts” and “longs”. This strategy becomes excessively prone to curve-fitting.

1 Like
#9
  1. What is so dangerous about over-optimization?

A strategy performing extremely well on a given dataset is likely to be “unready” to other market situations and conditions. Meaning, it is not general enough. Which might lead to missed profits or even loss of the funds.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • About 10 years to train the strategy
  • About 2 years outside that period to check how it performs on new unknown data.
  1. Why should you avoid asymmetric trading signals?

More probability of making a mistake. And more space for over-optimizing due to “more parameters to tweak”.

1 Like