Key Webinar Points:
Mistake #1: Not testing with enough historical data (at least 5-10 years)
Mistake #2: Not enough trades per rule.
How many rules and/or optimized inputs are there, which dictates degrees of freedom (ex: 10 trades with 10 rules would have 0 degrees of freedom; not good). The webinar included an example of a system that had 33 trades from 4 optimized inputs, which yielded 8 trades/rule, which is not generating enough trades per rule (i.e., likely over-optimization). He likes to see at least 30 (good), 50 (better), 100+ (best) trades per optimized input or rule. Not having enough rules per trade is a clear way to see if a trading system or product is being curve fitted or BS.
Mistake #3: Optimizing until "live" start
Do not optimize up to the current date, which leads to curve fitting and over optimization (oftentimes because not enough trades per rule were used). Kevin sees testing for a trading system like climbing a mountain, which includes an Out Of Sample test period ("OOS") (Good), Walkforward testing (Better) and live, real money testing (Best). He supports putting the strategy into an incubation period of 3-12 months to see how the trading system would have performed with your parameters.
Mistake #4: Not including (adequate) slippage
The previous system he showed was an overnight/swing strategy, so he showed how turning the system into a day system would change things (i.e., by adding in an end of day flat rule), however he did not include slippage (he did include commissions). Kevin always stresses that slippage and commissions be included in every strategy test. In previous webinars, he uses $5 for RT commission ($2.50/side) and 1-2 ticks for RT slippage (assuming spread is normally 1 tick). For instance, on the ES, the spread is typically 1 tick, and if he uses a market order in and out, then he assumes 2 ticks per RT ($25 ticks of slippage). Now in reality, his experience is that actual slippage is somewhere between 1 and 2 ticks (his experience is 1.5 ticks in slippage per RT) with ES market orders--HOWEVER IT GOES WITHOUT SAYING THAT THIS IS HIS EXPERIENCE AND SHOULD NOT BE TAKEN AS THE GOOD WORD. If one assumes limit orders in your strategy testing, whereby the testing engine will assume fills when the price touches the limit order, when in reality this doesn't happen in real life. Therefore, by including a 1-2 tick slippage per side on entries and exits, you are creating a fairly robust system. Therefore, when you include slippage and commission in your testing, parameter optimization will drive towards less trades (i.e., swing/overnight) rather than more trades (day). Including a healthy estimate of slippage and commission will create more robust systems. Be VERY cautious of vendor systems that do not include commissions and slippage in parameter selection.
Mistake #5: Tricking backtest engine
Trading software can be tricked to yield exciting, unrealistic, great looking equity curves. A good test is that you should create a strategy and then trick the backtest engine to create one of these unrealistic equity curves. By learning how to trick the engine, you will be in a better position to know what to avoid when testing you strategy. For instance, most trading software assumes that price moves from the Open to High/Low and then the Low/High and then Close...but as we know, the path price takes may or may not have done this. Be careful of limit orders, scalping strategies and exotic bar types--all of which can be invalidated by backtest engine results. However data can be an issue, too. For instance, the differences between closing (last traded price of the day) and settlement prices (established vary when they are recorded).
How can you tell if you've made one of these mistakes? (Check the "Taking a Trading System Live" thread for a more in depth exploration of this topic)
Method #1: Plotting walkforward, incubation and live equity curve. Looking for continuity between the periods.