Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I need sleep too (I drove home in middle of night last night after watching my Michigan Wolverines beat Notre Dame - biggest crowd in NCAA football history), but I'll try to keep up with you.
Tradestation I know is trying to work on better fitness functions. Most of the standard ones are weak. I usually use Net Profit, for simplicity, and typically, the good net profit cases have low drawdowns.
One other big issue is that using drawdown as part of a fitness function with walkforward can give very misleading results, if the walkforward window moves with time. I will try to explain when I answer an earlier post on walkforward.
I see the same stuff vendors are selling, and the worse thing is -- people are buying it. The other real problem with NinjaTrader's summary page is there is no position sizing anywhere. So you can make a system look great by just adding a few zero's to the position size, and it shows more trades, more profit. People then think it is not curve fitted because it has so many more trades.
The other big one is when people use limit orders. That's an easy way to make NT reports inaccurate.
I generally prefer systems that trade 1,000 times a year or more on 1 contract. If I start dipping below 500 trades a year, I get worried about over fitting. Obviously you take a very different approach, and this is just one of many ways we make a market together
I prefer systems that have very few input parameters, like 1 or 2 ideally, where everything else is dynamic and based on volatility and such in the market. That way I can optimize for my best score, and that golden looking equity curve we are all after, by combining that system along with many more into a portfolio and looking at how it benefits or balances the portfolio as a whole.
As for keeping track of the system, I wish I had fixed the code I was using to automatically post charts and trade results into a futures.io (formerly BMT) thread via the futures.io (formerly BMT) API. I just never had time to fix it, and certainly don't want to start now (I am preparing for vacation mode...). So I can't do any kind of live test. Closest I could do would be to shelf it and come back in two months and just run a new report to see how that out of sample data looks, but it would still be historical sim not live.
What ever came of their acquisition of Grail Computer? I thought they would be killing it by now with all the work Grail was doing. Have they integrated it into a production product yet?
Unfortunately, it is easy to over-fit even a large data-set. It is a matter of having enough degrees of freedom in your "solution" or what I am less familiar with, but have seen none the less is a simple MA crossing that gets lucky. (Likely large drawdown ,MAE, or ETDs)
From the "black magic" side, all you really need to do is give an algorithm enough time and degrees of freedom (input values, products/sums of terms, square's, square roots, etc.) and you can match to the max of degrees (ultimately more good trades) of freedom. Sometime it is difficult to see all the degrees of freedom, but there is likely more than you can see.
This is why Forward or Out of Sample testing is critical. The more "over-fit" a function is, the fast it will fall apart as you test values that were not used to train the system. What you are looking for is a "fit" function. One that does okay, but generalizes the solution into the future. This is the goal of my Genetic Programming investigation.
I do believe the Adaptrade Software is using Genetic Programming based upon configuration and function. I certainly cannot attest to the ability it has to solve the problem at hand.
As discussed above, it is very easy to "over-fit" a solution using any Machine Learning approach or (manually this is trickier but possible). I can tell you that I have many systems that will over-fit a solution to the training set. This is quite easy. The difficult part is to figure out how to get a ML system to create a Generic solution without access to the Out of Sample data.
This is a bit of a paradox for me right now. I want to create a solution that fits my training set (in sample data), but the second requirement is that it needs to also fit data I have not seen yet (Out of sample data). I can test the OOS data, but then it is touched and now should be considered "In Sample" data. It is unclear in systems like the Adaptrade system when it shows the OOS data on its charts, has it reviewed this data. It should be okay to calculate the answer (Profit or EC), but that data cannot be brought back into the Engine to affect the genetics of the system without it now being considered In Sample data again.
My current approach is to simply store anything that looks profitable. I then use a 2nd analysis which is manual for me to accept the strategy as generic. What I do is run all the "looks profitable" through NT (I use the optimizer, but only to load the different strategies for analysis) and check for nice EC curves and profitability OOS.
In the future, I may add more logic to truly analyse the "looks profitable" before storing. What may not be overly obvious is the longer you run a ML algorithm on a training set, the more likely it will specialize (over-fit) the training dataset. So the "looks profitable" strategies need to be pealed off as soon as they start to look good before over-fitting starts.
I have played with the Adaptrade software for a while. It was interesting, but not my cup of tea. In regards to the contamination of data set of OOS since it plots that in the chart. There is a thread in the googlegroup for them. As far as I understand it, it remains untouched as the results are not recycled. Basically all results that are used for the GP/GA optimization and next generation are based on the in sample. Then that goes to seed the next generation. The OOS results are shown but are not used to generate the next generation. But as a user you can see whats going on. You can go check the google group to be sure or ask questions that are answered by the programmer/founder of adaptrade himself.
@kevinkdog I find Adaptrades MSA is an excellent piece of software. If you have a solid system on 1 contract you can plug your test results in and really see how position size using different methods affects your outcome. Very powerful tool. Also has a nice portfolio feature and some other features that are good for analyzing a system. There a few things I wish it would do better but overall it is some of the best money I have ever spent. It is really inexpensive for what it does.
"The day I became a winning trader was the day it became boring. Daily losses no longer bother me and daily wins no longer excited me. Took years of pain and busting a few accounts before finally got my mind right. I survived the darkness within and now just chillax and let my black box do the work."
In an earlier post, I shared with you one way I track a trading strategy. Here is another tracking tool I use...
When I used to work in aerospace (or the "real world" as I sometimes refer to it), our small company ($250 million annual sales) would have a weekly meeting called "How We Doin." Incorrect grammar aside, it was an excellent way for the managers of the company to quickly see how sales were for the month and quarter, what quality and production problems were occurring, and just a general sense of where the company currently stood.
Now, fast forward a few years. I am trading full time, working alone. But, I still want to see at a glance "how I'm doin" with my strategies and trading. Obviously, my account statements and equity curve tell the overall story, but that is not enough detail for me. What strategies are doing good? Which are underperforming? Of strategies I am incubating, how do they look? Should I make some changes in what I am trading? This "How I'm Doin" report can help me answer all of these questions.
I developed a little spreadsheet (sorry, I am not sharing it, but it is easy enough to do yourself) to help me with this task. It tells me at a glance how my strategies are doing, and I can easily drill down and see detail if I need to.
First, there is a summary page. I include every strategy I am trading live. I also include in another section the strategies I currently am incubating. This summary sheet collects all the data I am interested in (of course, if you did this yourself, you'd likely pick different metrics than I did). This summary sheet gets the data from the individual sheets, which I will describe a bit later.
To keep things simple, I base everything on one contract being traded, even though that is usually not what I am actually trading. Why? My goal with this spreadsheet is to see how my strategies are doing compared to how I thought (calculated) they'd be doing. If I included position sizing, it would muddy up the view for me.
Of all the numbers on this sheet, I am primarily interested in two columns:
1. Return Efficiency - How am I doing, compared to my expectations? That is how I define return efficiency, and it is simply my actual return divided by my expected return. If my strategy is performing exactly as I had calculated, it will be 100%. Obviously, I want this to be close to or above 100%. Typically, when I take all the strategies together, I find my efficiency is somewhere between 70-100%. So, this says that if my historical testing says I should make $10 a year, I am actually making somewhere between $7-10.
2. Drawdown Efficiency - This is how I am doing with regards to drawdown. Just like with return efficiency, I calculate this as my actual drawdown divided by my expected drawdown. I then subtract the result from 1, to make the number 100% the ideal value. It is a bit backwards to do this, but I do it that way so that both efficiency numbers have 100% as their ideal value. Then, the closer to efficiencies get to zero, the worse off things are.
Once a month, I go through and update each of the individual system sheets with performance data, and that automatically updates the main sheet.