Here are results of an ES strategy that "failed."

I am not going to disclose the code, other than to say it would buy pullbacks in a long term uptrend, and vice versa for short trades. It made money on both long and short trades.

Development of strategy ended in mid 2015. Here is what backtested equity curve looked like:




For next 2 and a half years of live trading, it looked great...




But the the next 2.5 years were really bad, completely out of character for this strategy. Clearly a broken strategy at some point...



Just when you thought this strategy was destined for the scrap heap, the next year and a half were incredible!



And the last 1.5 years have crashed again.




Not sure this will help anyone, but this is a decent example of a good backtest, with a significant period (2.5 years) of good real time performance, followed by the strategy breaking...

kevinkdog 

... Not sure this will help anyone, but this is a decent example of a good backtest, with a significant period (2.5 years) of good real time performance, followed by the strategy breaking...

What interval was used to run this strategy?
Did you use a fixed time period if the interval was less than daily?
What was the logic for identifying a pullback ?

trendisyourfriend 

What interval was used to run this strategy? Did you use a fixed time period if the interval was less than daily? What was the logic for identifying a pullback ?

Daily


Price above longer term average, and also below a shorter term average

kevinkdog 

Here are results of an ES strategy that "failed." I am not going to disclose the code, other than to say it would buy pullbacks in a long term uptrend, and vice versa for short trades. It made money on both long and short trades. Development of strategy ended in mid 2015. Here is what backtested equity curve looked like: For next 2 and a half years of live trading, it looked great... But the the next 2.5 years were really bad, completely out of character for this strategy. Clearly a broken strategy at some point... Just when you thought this strategy was destined for the scrap heap, the next year and a half were incredible! And the last 1.5 years have crashed again. Not sure this will help anyone, but this is a decent example of a good backtest, with a significant period (2.5 years) of good real time performance, followed by the strategy breaking...

Good Morning Kevin,

Thank you for the example of your trading strategy displaying backtest and out of sample live trading. Many only show back test, noone never show what happen in reality.

May I ask you a question please?

1. You take the time and effort to back test or build a strategy/trading idea, that provided you an Edge per your criteria and liking. I am assuming you back test for over +10 years historical ES data and you love the equity curve. My question is why do you turn the Algo off after 2 years of running and say it is failure? Why you not run the algo forever or atleast 10 years live money because you back test for +10 years? I am confused by this, why turn of system after just 2 years?

Thank you,

And thank you for being a consistent trading educator in the trading business the past years helping traders.

goodoboy 

Good Morning Kevin, Thank you for the example of your trading strategy displaying backtest and out of sample live trading. Many only show back test, noone never show what happen in reality. May I ask you a question please? 1. You take the time and effort to back test or build a strategy/trading idea, which provided you an Edge per your criteria and liking. I am assuming you back test for over +10 years historical ES data and you love the equity curve. My question is why do you turn the Algo off after 2 years of running and say it is failure? Why you not run the algo forever or atleast 10 years live money because you back test for +10 years? I am confused by this, why turn of system after just 2 years? Thank you, And thank you for being a consistent trading educator in the trading business the past years helping traders.

Thanks for the kind words.

So to answer "why would I consider turning off an algo?" let's look at this strategy, assuming I traded 1 contract throughout...

If I had started real money trading right after development (something I do NOT recommend)...

First run up was +$25K per contract
Then a drawdown of about $18K
Then a 2nd runup of about $45K
Then a drawdown of about $25K
Finally a current recovery of about $8K

In summary, overall had you stuck with the strategy through thick and thin, you would have made $25K. BUT, you had to endure a long drawdown of $18K, and a shorter drawdown of $25K.

Could you have handled it emotionally?

I bet 99% of people could not.

Just imagine yourself in 2 scenarios:

1. You start trading around trade 68, you go on a cruise for years and years, and arrive home today. You never looked at the equity chart since trade 68. You'd probably be happy where you are at, with $25K profit.

2. You start trading around trade 68 and you look at equity curve every week. How would you feel at trade 105, when you have given back most of your profit? Would you be worried that your strategy was broken? I sure would. Nothing like that drawdown had ever happened in the backtest. Is that a fluke, or a warning sign?


When people look at hypothetical equity curves, they imagine themselves in scenario #1. But reality is scenario #2 - TOTALLY different feeling, for the same end results.



So why turn off a system? Well, remember the backtest is not a prediction of the future. You HOPE it continues, but "past performance is not indicative of future results." And imagine the psychological turmoil you'd experience when reality starts looking worse than the backtest (I love when people say "algo trading is emotionless," these sad souls are so misguided). This performance would have me pulling my hair out!

All things considered I think it is wise to plan for a different alternative than you see in backtest. So, you plan for failure, realizing it may never occur.


(By the way, this was a really good question. I plan on using this discussion in my next "algo Trading" article for Tech Analysis of Stocks & Commodities magazine).

kevinkdog 

So why turn off a system? Well, remember the backtest is not a prediction of the future. You HOPE it continues, but "past performance is not indicative of future results." And imagine the psychological turmoil you'd experience when reality starts looking worse than the backtest (I love when people say "algo trading is emotionless," these sad souls are so misguided).

Kevin has often said that traders who believe that automated trading will free them of emotional pressures and emotional trading are fooling themselves. He's never said it clearer, and with an excellent example.

Also, another gem: "Well, remember the backtest is not a prediction of the future."

Bob.

bobwest 

Kevin has often said that traders who believe that automated trading will free them of emotional pressures and emotional trading are fooling themselves. He's never said it clearer, and with an excellent example. Also, another gem: "Well, remember the backtest is not a prediction of the future." Bob.

Exactly. we're traders. Not crystal ball operators.

kevinkdog 

Backtests done correctly, though, can be meaningful. I recently did some research, running over 50 million backtests, across many futures markets, hundreds of strategies, multiple bar sizes, etc. - it literally took me months just to gather the data. The backtest is the In-Sample (IS) period, of around 10-15 years. The Out Of Sample (OOS) period is the immediate 3 years after the backtest. This is akin to when you'd be trading live, after the backtest period. I wanted to see if IS performance had any correlation to OOS performance. If it did not, then backtesting would truly be useless. My results: If the IS profit is unknown… 36.2% of backtests have profitable OOS performance (this would be like not running a backtest at all) If the IS profit is negative… 35.1% of backtests have profitable OOS performance If the IS profit is positive… 39.4% of backtests have profitable OOS performance In a nutshell, this says that backtesting can be useful. With a profitable IS test, we are more likely to have a profitable OOS result. So, this tells us that at least at a very high level, backtesting is worth doing. Obviously, this is not enough by itself to be useful in trading, but it is a decent start...

I don't understand this one: "If the IS profit is unknown… 36.2% of backtests have profitable OOS performance
(this would be like not running a backtest at all)"

How is the IS profit unknown if 32% of backtests had a profitable OOS?

phantomtrader 

I don't understand this one: "If the IS profit is unknown… 36.2% of backtests have profitable OOS performance (this would be like not running a backtest at all)" How is the IS profit unknown if 32% of backtests had a profitable OOS?

The backtest is the same thing as the IS performance.


"If the IS profit is unknown" ... I changed to "if you run no backtest"

kevinkdog 

The backtest is the same thing as the IS performance. "If the IS profit is unknown" ... I changed to "if you run no backtest"

Thanks for the explanation.

For the three test samples, the mean is 36.9% with no extreme standard deviation i.e. the results are about the same regardless whether you back test or not. I understand the sample is very large, but do you think the slight edge shown in the IS positive sample is significant?