NexusFi: Find Your Edge


Home Menu

 





NT8 Strategy Analyzer Problems


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one johns52 with 6 posts (2 thanks)
    2. looks_two koganam with 1 posts (0 thanks)
    3. looks_3 Fat Tails with 1 posts (2 thanks)
    4. looks_4 arni with 1 posts (1 thanks)
    1. trending_up 3,136 views
    2. thumb_up 5 thanks given
    3. group 6 followers
    1. forum 8 posts
    2. attach_file 2 attachments




 
Search this Thread
  #1 (permalink)
 johns52 
Michigan, USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
Posts: 18 since Jul 2014
Thanks Given: 17
Thanks Received: 6

NT8's Strategy Analyzer - I use commission, slippage, Tick Replay, COET - sometimes I get 12.5 profit factors and $30K per month in Total Net Profit. I only optimize for bar type and bar size. Then I run the strategy in Sim Trade on a chart - usually 15 minute - and steadily lose. What to do? The SA seems a quick way to evaluate a strategy when the market is running and I can't do Market Replay but what's the use if the results are so invalid? Is there a better way?

Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Business Collaboration Opportunity in Futures Trading
Traders Hideout
Futures Trading-New traders looking for solid PA trading …
Trading Reviews and Vendors
Cycle Through Ninjatrader Open Windows
NinjaTrader
Seeking Insights on Day Trading Options on Futures
Options
Multi-Data Strategy Questions
MultiCharts
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
The Program
9 thanks
Ask Me Anything (AMA) with NinjaTrader Client Services
2 thanks
Has anyone ever built any type of simple twitter scraper …
2 thanks
Daytrading ES & NQ
2 thanks
Blue wave trading review
2 thanks
  #2 (permalink)
 
Fat Tails's Avatar
 Fat Tails 
Berlin, Europe
Market Wizard
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
Posts: 9,889 since Mar 2010
Thanks Given: 4,242
Thanks Received: 27,107


johns52 View Post
NT8's Strategy Analyzer - I use commission, slippage, Tick Replay, COET - sometimes I get 12.5 profit factors and $30K per month in Total Net Profit. I only optimize for bar type and bar size. Then I run the strategy in Sim Trade on a chart - usually 15 minute - and steadily lose. What to do? The SA seems a quick way to evaluate a strategy when the market is running and I can't do Market Replay but what's the use if the results are so invalid? Is there a better way?

When there is a difference between the strategy analyzer and market replay, you need to analyze the differences trade by trade.
There are many potential sources for errors and discrepancies. You will only find out what is happening when you analyze it.

Typical problems:

- Use of exotic bar types which are not backtestable: What bar type do you use?
- Use of fast bars (high frequency trading) such that many bars have the same time stamp and orders can not be properly executed.
- Use of other than market orders without selecting the correct order fill resolution or adding a secondary bar series to the strategy.

As there are many potential sources of errors, you need to give more information on your strategy, such as

- bar types used
- bar period used
- order types used

In most cases, when a strategy backtest goes wrong, it is the result of someone having committed a few mathematical crimes. :-)

In case you do not want to share further information, just run a backtest over a short period of time and examine the trades one by one, comparing the backtest and the market replay.
Also you may want to strip down the strategy to a simpler version and then go step by step adding features and look whether it has an impact on the discrepancy between backtest and market replay.

Reply With Quote
Thanked by:
  #3 (permalink)
koganam
Garner, NC/USA
 
Posts: 39 since Dec 2009
Thanks Given: 16
Thanks Received: 42



johns52 View Post
NT8's Strategy Analyzer - I use commission, slippage, Tick Replay, COET - sometimes I get 12.5 profit factors and $30K per month in Total Net Profit. I only optimize for bar type and bar size. Then I run the strategy in Sim Trade on a chart - usually 15 minute - and steadily lose. What to do? The SA seems a quick way to evaluate a strategy when the market is running and I can't do Market Replay but what's the use if the results are so invalid? Is there a better way?

Is your strategy running on Renko bars, or any other kind of synthetic bars, by any chance?

Reply With Quote
  #4 (permalink)
 johns52 
Michigan, USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
Posts: 18 since Jul 2014
Thanks Given: 17
Thanks Received: 6

Hello, thanks for your responses. I am generally using minute bars. I always use market orders as I want to be sure and get a fill. I should have had all my information lined up in advance so to catch up I ran a study today. It's a simple Donchian Strategy with no stops or profit targets: I think it was the stops & profit targets hurting the prior results. I use Low3Ago (Low[3]) to exit the longs and High3Ago (High[3] to exit the shorts. I usually run the backtest on 15 min, then optimize from there. Attached are today's results. The time periods are a little weird since I started with 5 minutes and advanced by 10s. I was surprised to see that longer time frames gave better results. I have no way to determine the drawdowns other than go through the trades with the chart's Strategy Performance/Historical. I assume longer time frames have larger drawdowns.

The Strategy Analyzer for the 2nd best result (115 min) give $66K for the 6/1-8/22 period with tick replay, commission and slippage of 1. Market Replay gave +$12.7. (Great though).
The SA for for the 5th best result (65 min) gives $71.3K - MR was +$4.1K.
The SA for the 10th best result (35 min) gives $91K - MR was +$13.9K.
The SA shows the best results for a 95 min period with +$75K. Unfortunately, the Market Replay stalled out after 6 days. I will have to run it again.
Well, this looks promising and gives me some hope. A little more tweaking and a lot more testing and I may be ready to try it.

Is there any reason why I can't run 4 ES charts at once with the same strategy? Why would one stall out on me?

Could the Strategy Analyzer software be skewed to be wildly optimistic?

Does running Market Relay at 1000x hurt the results?

Strategy Analyzer Donch3 8-23-2022


Market Replay ES 6_1 - 8_22 4 times 1000x

Started this thread Reply With Quote
Thanked by:
  #5 (permalink)
 johns52 
Michigan, USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
Posts: 18 since Jul 2014
Thanks Given: 17
Thanks Received: 6

from the NinjaScript guide: https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?discrepancies_real-time_vs_bac.htm


Quoting 
You should expect that a strategy running real-time (live brokerage account, live market simulation, Playback connection etc...) will produce different results than the performance results generated during a backtest.

I guess that says it all: Strategy Analyzer will not give the same results as real-time or playback. That's a big disappointment.

Started this thread Reply With Quote
  #6 (permalink)
 johns52 
Michigan, USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
Posts: 18 since Jul 2014
Thanks Given: 17
Thanks Received: 6

from the NinjaTrader Forum: https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance

Quoting 
Comparing real-time, historical, and replay performance
07-23-2017, 05:12 PM
Citizens of the NinjaTrader Community,

A common question we hear from clients is 'why are results from backtest different from real-time or from market replay?'.

The main reason is due to a lack of intra-bar granularity.

See more about intra-bar granularity.
https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377

By default when backtesting no intra-bar granularity is added. An added 1-tick series is necessary to increase fill accuracy to within 1 tick.
Further, TickReplay is necessary for intra-bar indicator values to update and recalculate for on each tick or on price change Calculate setting.

As a heads up, TickReplay cannot be used for accurate order fill prices and High Order Fill Resolution cannot be used in conjunction with TickReplay.

Apparently my question is very common. In my case, I am getting fantastic results with COET and Tick Replay and an added 1-tick data series. It's when I go to Market Replay/Playback that the 12.5 profit factor drops to 1.1 and the drawdowns go from $650 to $6500.

Wish I knew how to rectify that. Is it slippage? Do I need to use a Stop-Market order? Why are entries & exits happening a bar late?

Started this thread Reply With Quote
  #7 (permalink)
 johns52 
Michigan, USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
Posts: 18 since Jul 2014
Thanks Given: 17
Thanks Received: 6

Starting to figure this out.....

Here's a very helpful video from Chelsea B:


Here's the Forum page: https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance

Zip file for the RealtimeReplayHistoricalComparisonsExample_NT8.zip : https://forum.ninjatrader.com/filedata/fetch?id=908317

Started this thread Reply With Quote
Thanked by:
  #8 (permalink)
 johns52 
Michigan, USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: futures
Posts: 18 since Jul 2014
Thanks Given: 17
Thanks Received: 6

My tests suggest that Tick Replay must also be used with Market Replay to get closer to IRL results. Try it - you'll see huge differences between with and without Tick Replay. My tests also suggest you will need to have a 12.5 profit factor or better in the Strategy Analyzer to get a possible 1.5 profit factor in Market Replay/Tick Replay.

Here's a tip: for a Market Replay from 8/1 - 8/22, set Days to Load in the Data Series to 22. They you can access Strategy Performance >> RealTime and get the Trades, Max Drawdown, Profit Factor, etc.

I am learning all this by drips and drops. Does someone teach this? Kevin Davey works with Tradestation which is a much simpler system; Easy Language not C#.

Started this thread Reply With Quote
  #9 (permalink)
arni
Hamburg, Germany
 
Posts: 5 since Dec 2021
Thanks Given: 1
Thanks Received: 2

Hi,

I have also run into problems with NT8 backtesting vs real-life trading. If you run your backtest with Order fill Resolution = Standard (fastest), there is a strange calculation being used in the background which leads to the results being just wrong. Instead, set the Order fill Resolution = High, then 1 tick for highest accuracy. Definitely takes much longer to run the backtest, but your results will be as close to real trading as possible. Thats actually what they replied to me in an email when I asked them about it.

Reply With Quote
Thanked by:




Last Updated on September 27, 2022


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Sitemap - Downloads - Top
no new posts