Strategy Creation

The Pitfalls of Backtesting – How to Avoid Overfitting and Disappointment

Matt Young May 20, 2021

Table of Contents:

  1. What is overfitting?
  2. How do I avoid overfitting?

For the investors reading this article, you can skip ahead to “What is Overfitting?”

By this point you may have run a few backtests, on a simple script or using one of our educational scripts. You may have also achieved a very high Profit/Loss % and are wanting to run the strategy with real assets. Before this happens, I want to caution you and help set expectations.

What is overfitting?

If an analysis corresponds too closely with a limited set of data we call this overfitting. Markets inherently have noise, that is to say small and frequent idiosyncrasies in the price data. When modelling a strategy, we want to avoid optimizing for a specific period because there is a chance this won’t predict the future. 

It’d be like tuning a car specifically for one racetrack, while expecting it to perform well anywhere.

To illustrate:

In the example above, 100 versions of the default script were backtested for 5 months. One of which yielded over 93% profit. If an imaginary user went live for the next 5 months using these ‘best’ parameters, they might be disappointed. Nobody ever complained about more money, but 12.7% is notably less when compared to the results of the previous 5 months. The point is, the past doesn’t necessarily predict the future.

How do I avoid overfitting?

1. Test for random periods

Running a backtest during bull and bear markets is a good first step. However, the reality is you are probably not starting a strategy at the beginning of a new market cycle, but in the middle of one. Sideways markets can be especially frustrating as a strategy tries to decide bear and bull trends. How would you react if your strategy lost money for the first 3 months?

2. Be granular in your testing

Due to market volatility, it’s possible that one good month (or one good trade) is enough to offset a large loss. Profitable trades and max drawdown MDD offer a window to overall performance, but does not tell a detailed story. To help, we’ve made our own Power BI Template that organizes trade data for deeper analysis.

3. Run a Paper Trade (forward test)

This one’s more of a reminder. 

I want to end by saying that these methods are not comprehensive. Each strategy is different, making it hard to offer blanket advice. The point of this article was to raise awareness.


Tags:

Disclaimer: The opinions expressed here are for general informational purposes only and are not intended to provide specific advice or recommendations for any specific security or investment product. You should never invest money that you cannot afford to lose. Before trading using complex financial products, please ensure to understand the risks involved. Past performance is no guarantee of future results.

All investing involves risk, including the possible loss of all the money you invest, and past performance does not guarantee future performance. Historical returns expected returns, and probability projections are provided for informational and illustrative purposes, and may not reflect actual future performance.

Tuned is not a broker-dealer, exchange, custodian, wallet provider, or counterparty. Investing is highly speculative and volatile. Tuned is only suitable for investors who fully understand the risk of loss and may experience large drawdowns. Investors should never invest more than they can afford to lose.