Reading Assignment: Common Backtesting Mistakes


#1

In this reading assignment we will dive deeper into the most important issue when it comes to building a profitable trading strategy - Backtesting. We will look at some of the common mistakes, over-optimization and curve-fitting and how you can avoid them. Read through this blog post and answer the following questions in this forum thread. Use the knowledge you have learned so far as well.

  1. What is so dangerous about over-optimization?
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
  3. Why should you avoid asymmetric trading signals?

#2
  1. curve fitting; it maybe over adjusted or tailored to past events (back testing) and may not be suitable for the future markets or new future data sets.

  2. snapshot 6 months; with a comparison to the longest possible (>5 years ideally, 9 to 11 years) data set humanly possible!

Focus on the development of a trading system aiming for adaptability, broad optimizations, robust profitability and large periods of testing data increasing better chance or probability of achieving a Ferrari.

  1. Asymmetric system; Asymmetric information can lead to adverse selection, incomplete markets and is a type of market failure.

New Algo Team; join the A-Team.

Serious about Algo optimisation keep in touch & say hello; opentrade@protonmail.com


#3
  1. Over optimization can lead to “curve-fitting”, which is the unwanted tuning of the strategy to fit specific past data.

  2. According to the article, approx 10yrs on higher time frames for accuracy.

  3. Asymmetric signals provide more complexity which tends to lean towards tuning the strategy for specific datasets… curve-fitting.

Funny I haven’t heard of this term “curve-fitting” before, but I’ve certainly been victim of it while fine tuning strategies in the past. Interesting ;o)


#4

What is so dangerous about over-optimisation?

  • There is a danger of over-optimisation is to produce a trading strategy with absolutely astonishing results that will not be achievable going forward.

How long should a testing period be if you are serious about building a profitable trading strategy?

  • Time frames should be greater than 30 minutes
  • Ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.

Why should you avoid asymmetric trading signals?

  • Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitting solutions.

#5
  1. What is so dangerous about over-optimization? Over-optimization is dangerous because it trasforms the strategy in a curve-fitting of past data that will be never identic to the future data. So the very good results on the past data will not reply in the future.

  2. How long should a testing period be if you are serious about building a profitable trading strategy? In the traditional market a testing period of 9-11 years should be used.

  3. Why should you avoid asymmetric trading signals? Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.


#6
  1. What is so dangerous about over-optimization?
    Creating a false sense that plan will work just customized for past data.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    About 10 years

  3. Why should you avoid asymmetric trading signals?

You end up matching up with previous market cycles.