Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
100% agree on the pitfalls of over optimization & curve fitting but also 100% disagree when it comes to saving time testing and discounting strategies that simply don't work out of the box and need tweaking/filtering/etc. to be worthy of consideration. What I've finally learned after almost 20 years trying to automate my trading to be 100% mechanical is that bar-centric strategies can never developed to be reliably used statically (i.e. without some sort of auto-adjusted parameters).
Sure, leveraging proper money management is always important to keep expectancy at acceptable levels, but I still have found it very important to be able to back test as many ideas as possible - as QUICKLY as possible. Ultimately, after testing hundreds+ of strategies over the years (both my own and ideas picked up elsewhere), my go to "back of tricks" has settled to only ~5 that are consistent enough for actual trading.
I'm not really a fan of any GC-based language/framework which is one of the reasons I discounted MotiveWave years ago. Java has been made considerably faster over the years but it will never be as fast as C++ or Rust though it hopefully won't need to be. Given that I'm currently trying to process individual ticks & order flow in real time, performance has become more critical for what I'm developing; whether MW will suffice is still TBD but I'm giving it a strong second look.
I've been impressed at how quickly MW order flow charts are constructed for NQ while monitoring TradeStation on a different monitor FWIW.
I chose Qt since it was the best supported GUI API for Python, but Python itself wasn't up to the task (performance-wise; Mojo will ideally fix that at some point but it's still deep in development-mode). While I've used C++ a while ago, coding an entire app in that would have taken me considerably longer and I 1) wanted to learn Rust (for career reasons) and 2) the best way to learn a new language is to try porting an existing app. I've managed to get about 80% of the port completed at this point...
Original Python-based UI
Current Rust-based UI
NinjaTrader lost me at lack of real-time replay, otherwise I've found NT8 to be pretty decent (and coding in C# isn't that horrible). TradeStation for that matter doesn't have replay functionality at all quite frankly but OOEL is considerably better than MC PowerLanguage (though I have thousands of lines of EasyLanguage that work on both platforms). The fact that I also had to convert my actual strategy to work as an indicator in TradeStation (for reasons I won't go into here) means I can't even use TS backtesting and had to roll my own.
Eubie
To end positively, writing my own app suite was definitely exciting and if you’re planning on doing it, go for it. I wish I had had someone to write mine with, as I would probably have avoided some unnecessary coding (like adding support for options) and some dead ends that only cost me a ton of time/money (like anything related to iqfeed mbo data, what a bunch of losers they are). If you can get a coding buddy, that’s gonna help a ton.
I should probably cancel my IQFeed MBO subscription as I haven't bothered to dip into the extra levels of order book yet - thanks for the reminder. Whether I need to finish my Rust app remains to be seen as it's taken me over a year to get it to this point. Qt charting isn't great, but since I have C++ source I could probably extend it to do the few things that are lacking (that a lot of others are also looking for as well). Getting Rust to actually use Qt is another rabbit hole with limited options available so I'm hoping MotiveWave solve things in the near term for now.
Eubie
EDIT: This was not my best-articulated post, so I will try to sum it up
1/ congrats on deciding to write your own trading apps
2/ you want to make this as easy to do as possible because motivation will run out before you see some applicable results of your work
if I were you, I would cut the nice to have features and didn't think about them until you felt the need to use them
from my point of view, this includes the ability to edit EL unless you're familiar with parsing, tokenizing, compiling ... etc
from my point of view, this also includes support for OSes that you don't use yourself. Supporting them increases work for you while the reward for your is questionable - the good feeling that you have a killer app that supports Mac will I think not keep you going
3/ don't focus on runtime performance, most compiled languages that have some perf part in them (e.g. Unsafe and Span in C#), are pretty much in the same perf league and as a user you will not notice a +/- 5% runtime difference. Choose a language that you're cool with as this will likely be a 1-2 year project and complicating it by learning Rust first is going to make it a 2-3 year project just to make it 5% faster in a good case scenario
At the beginning, I wanted to have all the features coded perfectly. When an idea for a new feature came, I couldn't get it out of my head until I coded it. Now, some two years and 80KLOC later, I am wondering why I was coding equity options split adjustment, when I wanted to predominantly focus on futures I want you to avoid the mistakes I did. Cheers!
Thanks for taking the time to respond - much appreciated and plenty of food for though from a fellow "roll your own" trading app developer!
As a 40+ year IT professional, I've spent a considerable amount of time (past 18+ years) trying to mesh my passions for both trading and programming, initially as a hobbyist and now full-time. I've used most of the popular retail trading platforms ( NinjaTrader, MultiCharts, MotiveWave and for the past few years, mainly TradeStation 9.5 & 10) and have had many love/hate experiences with all of them. I have life-time licenses for a few though am personally against paying via any subscription-based method so that has kept Sierra Chart off my list (though I did try it and was impressed but even it lacked some of what I've found to be missing from all of the others to some degree).
My personal wish list of platform requirements (basically what TradeStation 10 currently has now PLUS):
Real-time historical tick playback (I should be able to play back ticks for a given day at whatever speed I want, including exactly how it occurred, not just some arbitrary "10 updates per second" nonsense)
The ability to simply click on a bar to specify a replay starting point (MultiCharts has this but their real-time playback is still lacking)
Optimization engine with cached indicator results that can be reused throughout the optimization process - this speeds up optimization insanely well though I'm not sure if any retail platforms actually do this
Access to tick-level Bid/Ask "aggressor" data as provided by IQFeed but never exposed at the (EasyLanguage) API level (TradeStation & IB data aren't close to IQFeed for a number of reasons)
Accurate Renko bars with high/low wicks (TradeStation fails on this front, causing backtest results to be over-optimistic)
Ability to update indicator parameters "live" and see the effect on the UI immediately (think discretionary "tuning")
Support for Windows, Linux and OSX
Mind you, there's a LOT of what TradeStation offers that I'll never use but others might - it blows my mind though why real-time replay just isn't considered "important" (or maybe it's TOO important - hmm). There have been some hacks mentioned in the TS forum that I've tried but they're just that - hacks.
As mentioned elsewhere in this thread, it should be fast, accurate AND reliable (not just the usual "pick any two"). It should have an event-based, multithreaded architecture and access to some sort of scripting language is assumed available. I've grown accustomed to EasyLanguage (OOEL) in particular as I can extend it via DLLs as needed (using whatever language I care to develop in) and I've found I can do a lot more with fewer lines of code in EL. In a perfect world, I'd be able to debug OOEL via VSCode (not just edit), alas, that probably will never happen. My panacea open source platform would translate and compile EasyLanguage/OOEL indicator & strategy code to LLVM and let me run at maximum speed like most retail platforms do today already - someday...
As Mike and others have mentioned, there are a few of us here who decided to bite the bullet and try to roll their own platform as there simply isn't an open source option that I've ever seen that comes remotely close to the commercial retail platforms. If all you're looking to do is relatively simplistic bar-level strategies (MA crossovers, momentum-based, etc.), you probably don't need what I'm looking for which extends beyond that into trying to automate discretionary tick-level price action activity that my eyes can see and process but I've yet to convert 100% to code. Despite having spent years trying literally thousands of bar-level strategies, I've never been able to consistently match what I can do manually using price action alone which is somewhat more difficult to wrap in some algorithmic "secret sauce" (at least for me). Attempts at leveraging machine learning made me realize even that was more about having the right feature sets up front; "indicator soup" is only half (at best) of the data required. so plenty more investigation is still TBD on that front.
Since I also want to be OS-agnostic, I initially thought of seeing what I could do with Python alone and developed Fu510nTrader but ran into performance issues that forced me to stop and re-think architecture. Were Mojo more mature and available for native Windows development (and had support for Qt6), I might have gone that route alas that's probably at least a year+ away from reality. Still, I learned a lot along the way, particularly on UI best practices when writing Qt applications. The platform does do everything on my wish list but has many gaps to fill before ever being able to use for live trading. I decided to tackle the performance issues by porting the whole thing over to Rust last year, which is 100X-1000X faster for sure but here again, developing anything in Rust has a considerable (vertical?) learning curve which took me a year to port and it's still very much "in progress" as a PyQtGraph equivalent really doesn't exist for Rust and Qt's charting API leaves a lot to be desired. Charting alone is a rat's nest of it's own which is making me think I should really just try to integrate with one of the JavaScript-based frameworks like Highcharts or ChartIQ (the former can be used for personal projects).
Which is all just a long-winded way of saying - it's a LOT of work, especially for one person (or even a small group) to attempt alone but having invested several thousand hours in the code I've written over the years, I can see how/why most are reluctant to simply "give it away" and have others potentially profit from one's efforts. I'm not sure really where I stand at this point as I have a few remaining milestones that keep me 150% busy most of the time these days. Packt did reach out to me last year to write a book on mastering EasyLanguage language on TradeStation that I turned down for a few reasons (lack of time be the primary one) though I think we'd all really benefit if there were a FOSS trading platform that literally "did it all" the absolute best way possible and, like Linux, blossomed well beyond what Linus initially started out with.
Thoughts?
-Guy
FWIW, I asked ChatGPT for "an itemized list of all the features in TradeStation 10 (do NOT summarize, but list all of them)" just to see what it would come up with:
1. Advanced Charting Tools
Interactive Charting Interface
Multiple Chart Types (e.g., candlestick, bar, line)
46. Support for Dividend and Corporate Action Tracking
Dividend Schedules
Stock Splits and Mergers Notifications
Corporate Action Updates
47. Integration with Tax Optimization Tools
Tax-Loss Harvesting Features
Capital Gains Tracking
Deduction Calculators
48. Comprehensive Order History Tracking
Detailed Logs of All Orders
Searchable Order History
Order Status Tracking
49. Customizable Workspace Layouts
Save Multiple Layouts
Quick Switching Between Layouts
Layout Sharing Options
50. Support for Alternative Data Sources
Social Media Sentiment Analysis
Geospatial Data Integration
Satellite Data for Market Insights
...asking it to produce a high-level PlantUML diagram describing TradeStation 10 produced (food for thought):
I'd also like to add:
Backtest & optimization engines that provide configurable starting & ending times in addition to a jump/skip feature for jumping forward to the next day if daily profit/loss targets have been hit (vs. looping over ALL ticks/bars unnecessarily when backtesting weeks/months of data; this shaves a LOT of wasted time)
Optimization engine with configurable # of threads to use
Backtest & optimization engines that provide configurable starting & ending times in addition to a jump/skip feature for jumping forward to the next day if daily profit/loss targets have been hit (vs. looping over ALL ticks/bars unnecessarily when backtesting weeks/months of data; this shaves a LOT of wasted time)
Optimization engine with configurable # of threads to use
I see you mean to do this project for real
I looked more into Rust and I understand the appeal, though I am really curious how much quicker the runtime is going to be. I know this aint your main reason to use it, but still.
As for IQFeed MBO ... here's my experience with them in a couple of sentences
Jan, 24
Me - Guys, you have serious bugs in the feed, resulting in the ES book being crossed multiple times a day for seconds/minutes.
Support - We can't investigate.
Me (writing to somebody up the chain) - Support says they cant investigate a brutal error in your api.
Support - Oh ... this was probably a misunderstanding, we meant ... we need you to send us a logfile etc.
Me - Here's the logfile, here's the grep of all the relevant lines where an order remains on the book even though it's been traded away seconds ago (so the delete operation related to a trade is missing).
Support (a week later) - Thanks, we have identified a bug, where some orders are deleted quicker than they should (so there are more deletes than necessary)
Me - I reported an entirely different bug, how can you say a missing delete and an extra delete are the same thing??
Support - No response.
Based off of this, I deleted 30 TB of realtime MBO data from IQfeed, after spending a week trying to repair it. 16K USD just made "poof".
Mar, 24
Me - You havent responded, I dont understand how you can be so incompetent, all the new feeds like Polygon and Databento are going to eat your lunch before you say iqfeed.
Support - No response.
Jan, 25
Support - We would like to inform you, that in the next release of our client, the bug you reported will be fixed! Below, you will find a description of the bug, whereby an update of an order doesnt report an mbo delete, when conditions XYZ happens.
So all in all, it took them a YEAR to fix a bug that I didn't even report and the ones that I reported are still in the feed. It's kinda sad that IQFeed is really best only for historical tick data, which it was the best for some twenty years ago already.
I looked more into Rust and I understand the appeal, though I am really curious how much quicker the runtime is going to be. I know this ain't your main reason to use it, but still.
The Rust runtime (being LLVM-based), is one of the "best on all fronts" (IMHO) compilers/frameworks with which to build just about any application. There are plenty of languages that use LLVM, most recently Mojo, that I'm really looking forward to once it becomes available natively for Windows development. The Rust learning curve is steep for sure, GUI support is somewhat lacking (but getting better) and it's definitely NOT something one becomes proficient with in a couple weeks. There are facets of OO that C++/Java have that Rust doesn't that I miss but it also tends to make things a bit simpler coding in a more functional way. That being said (and Keven tried to alliterate but nuked his message(s) ), I'd REALLY rather not waste time building my own platform - it's been a fun/interesting experiment from a hobbyist perspective, but posting my "want list" was simply my way of even reminding myself why I'd even bother trying to build something from scratch if other platforms had what I need. I've already "invested" all the typical time others expend learning the ropes over the past 18+ years, donating enough to understand the importance of actually having an edge, money management, backtesting all the published and my own strategies, etc.
It wasn't until I became relatively proficient at price action trading, I realized that all the bar-level strategies I developed over the years would never be profitable enough unless it took price action into consideration whether for signal generation or filtering. It was at that moment that real-time playback (with sub-second timestamps, etc.) became my #1 must have - something that exists in a few platforms for sure, but not without some of the other limitations I previously mentioned. I'm spending some time now revisiting MotiveWave since it does check quite a few boxes (most notably real-time playback) that will let me at least create the algorithm side of things that I'll ideally back-port to TradeStation anyway.
Quoting
As for IQFeed MBO ... here's my experience with them in a couple of sentences
Jan, 24
Me - Guys, you have serious bugs in the feed, resulting in the ES book being crossed multiple times a day for seconds/minutes.
Support - We can't investigate.
Me (writing to somebody up the chain) - Support says they cant investigate a brutal error in your api.
Support - Oh ... this was probably a misunderstanding, we meant ... we need you to send us a logfile etc.
Me - Here's the logfile, here's the grep of all the relevant lines where an order remains on the book even though it's been traded away seconds ago (so the delete operation related to a trade is missing).
Support (a week later) - Thanks, we have identified a bug, where some orders are deleted quicker than they should (so there are more deletes than necessary)
Me - I reported an entirely different bug, how can you say a missing delete and an extra delete are the same thing??
Support - No response.
Based off of this, I deleted 30 TB of realtime MBO data from IQfeed, after spending a week trying to repair it. 16K USD just made "poof".
Mar, 24
Me - You havent responded, I dont understand how you can be so incompetent, all the new feeds like Polygon and Databento are going to eat your lunch before you say iqfeed.
Support - No response.
Jan, 25
Support - We would like to inform you, that in the next release of our client, the bug you reported will be fixed! Below, you will find a description of the bug, whereby an update of an order doesnt report an mbo delete, when conditions XYZ happens.
So all in all, it took them a YEAR to fix a bug that I didn't even report and the ones that I reported are still in the feed. It's kinda sad that IQFeed is really best only for historical tick data, which it was the best for some twenty years ago already.
So, when are you gonna start?
Thanks for the further MBO heads up - since it's already pricy it doesn't make sense to keep paying for "garbage data" (especially since I'm not using it ATM anyway).