The real dilemma here is you have 2 sets of historical performance results:

1. Your strategy, presumably developed with your backtest approach
2. Strat #2, developed by someone else, in an unknown way

Maybe one or both are the results of optimized backtests, maybe they are live trading, maybe in-sample, maybe out-of-sample. Maybe they are real money results. Who knows?

@USIndexTrader might know the answer, at least for their strategy.


Unfortunately, you really cannot tell much from the performance metrics when the source of the data is unknown. Most people could, for example, create a backtest with metrics 10x better than the ones shown here. But it would not mean it actually was a better strategy. It could fall apart in live trading. It would be a better backtest, but that's all.

I hope you see the point. The real comparison you should be making is in HOW those results were obtained. (which for subscription strategy you may never know)

Think of it this way:

Maybe your strategy was developed the same way you've developed 100 other strategies, most of which performed well in real time (that would be great!)
But maybe your strategy is a an optimized backtest, your first attempt at strategy building, and all results are in-sample (that would be bad)

Maybe the subscription strategy was created yesterday by a developer desperate to get subscription money, and he has never traded live (bad, obviously)
Maybe the subscription strategy was created by the late trading legend Jim Simons (that would be good)


I realized I've just completely muddied the waters by focusing on the HOW of strategy creation, as opposed to the WHAT they produced. Sorry for that. But I think that is more important than any metric.

Monte Carlo analysis is nice, but it doesn't help in this situation. If you feed it curvefit results, it will give great output. It doesn't know if the data is garbage to begin with.


By the way, speaking of subscription based strategies, here is a really great and under appreciated study by one of the best real money traders on this site (in my opinion):

@SMCJB did a lot of work here, it is worth a look:

iSystems Journal Greetings and welcome to my iSystems Journal. While most people seem to create journals to help them with the psychological aspects of trading, the intent of this journal is to document my experiences with isystems.com in the hope that it may be helpful …

kevinkdog 

The real dilemma here is you have 2 sets of historical performance results: 1. Your strategy, presumably developed with your backtest approach 2. Strat #2, developed by someone else, in an unknown way Maybe one or both are the results of optimized backtests, maybe they are live trading, maybe in-sample, maybe out-of-sample. Maybe they are real money results. Who knows? @USIndexTrader might know the answer, at least for their strategy. Unfortunately, you really cannot tell much from the performance metrics when the source of the data is unknown. Most people could, for example, create a backtest with metrics 10x better than the ones shown here. But it would not mean it actually was a better strategy. It could fall apart in live trading. It would be a better backtest, but that's all. I hope you see the point. The real comparison you should be making is in HOW those results were obtained. (which for subscription strategy you may never know) Think of it this way: Maybe your strategy was developed the same way you've developed 100 other strategies, most of which performed well in real time (that would be great!) But maybe your strategy is a an optimized backtest, your first attempt at strategy building, and all results are in-sample (that would be bad) Maybe the subscription strategy was created yesterday by a developer desperate to get subscription money, and he has never traded live (bad, obviously) Maybe the subscription strategy was created by the late trading legend Jim Simons (that would be good) I realized I've just completely muddied the waters by focusing on the HOW of strategy creation, as opposed to the WHAT they produced. Sorry for that. But I think that is more important than any metric. Monte Carlo analysis is nice, but it doesn't help in this situation. If you feed it curvefit results, it will give great output. It doesn't know if the data is garbage to begin with. By the way, speaking of subscription based strategies, here is a really great and under appreciated study by one of the best real money traders on this site (in my opinion): @SMCJB did a lot of work here, it is worth a look: iSystems Journal Greetings and welcome to my iSystems Journal. While most people seem to create journals to help them with the psychological aspects of trading, the intent of this journal is to document my experiences with isystems.com in the hope that it may be helpful …

Kevin, thank you for your insight. It’s always invaluable as usual!

USIndexTrader 

Kevin, thank you for your insight. It’s always invaluable as usual!

I'm curious, how did this all turn out? What was your conclusion with those strategies?

Hi Kevin - I stopped trading the paid for system as I wanted to concentrate on my own system. I am in the process of testing and validating my system again. Letting it trade in SIM currently to see how it performs. Last week it lost $545, all other weeks in May were profitable. Needs more time to incubate to draw firm conclusions.

USIndexTrader 

Hi Kevin - I stopped trading the paid for system as I wanted to concentrate on my own system. I am in the process of testing and validating my system again. Letting it trade in SIM currently to see how it performs. Last week it lost $545, all other weeks in May were profitable. Needs more time to incubate to draw firm conclusions.

Thanks for the update, hope you have success.

Was performance an issue in stopping that paid system, or was it the control you have over yours , etc?

kevinkdog 

Thanks for the update, hope you have success. Was performance an issue in stopping that paid system, or was it the control you have over yours , etc?

100% control. And you hit on a good point being that I don't know how the paid system was developed. That was a helpful piece of insight.

USIndexTrader 

100% control. And you hit on a good point being that I don't know how the paid system was developed. That was a helpful piece of insight.

There is nothing like 100% knowing how a strategy was developed. When you do this with a strategy building process that you know works, your confidence will soar!

Good Luck!

Kevin