Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Is there any interest in a unix based polyglot trading toolkit? I've been developing a set of tools in ruby that's CLI based. Each component is split up into a separate process so they can be swapped out with a different implementation e.g. currently the backtester is ruby which is fast enough for the stuff I'm doing but could easily see how someone might want to write it in rust, c, etc...
I was hoping I might be able to find some like minded people on these forums so that we could form some sort of collaboration.
Can you help answer these questions from other members on NexusFi?
I have been thinking of building open source vendor agnostic financial API for market data and order execution for about a year or two now. I finally decided to do it. The result is ZMAPI. It has fully functional market data (MD) support …
As a 40+ year IT professional, I've spent a considerable amount of time (past 18+ years) trying to mesh my passions for both trading and programming, initially as a hobbyist and now full-time. I've used most of the popular retail trading platforms ( NinjaTrader, MultiCharts, MotiveWave and for the past few years, mainly TradeStation 9.5 & 10) and have had many love/hate experiences with all of them. I have life-time licenses for a few though am personally against paying via any subscription-based method so that has kept Sierra Chart off my list (though I did try it and was impressed but even it lacked some of what I've found to be missing from all of the others to some degree).
My personal wish list of platform requirements (basically what TradeStation 10 currently has now PLUS):
Real-time historical tick playback (I should be able to play back ticks for a given day at whatever speed I want, including exactly how it occurred, not just some arbitrary "10 updates per second" nonsense)
The ability to simply click on a bar to specify a replay starting point (MultiCharts has this but their real-time playback is still lacking)
Optimization engine with cached indicator results that can be reused throughout the optimization process - this speeds up optimization insanely well though I'm not sure if any retail platforms actually do this
Access to tick-level Bid/Ask "aggressor" data as provided by IQFeed but never exposed at the (EasyLanguage) API level (TradeStation & IB data aren't close to IQFeed for a number of reasons)
Accurate Renko bars with high/low wicks (TradeStation fails on this front, causing backtest results to be over-optimistic)
Ability to update indicator parameters "live" and see the effect on the UI immediately (think discretionary "tuning")
Support for Windows, Linux and OSX
Mind you, there's a LOT of what TradeStation offers that I'll never use but others might - it blows my mind though why real-time replay just isn't considered "important" (or maybe it's TOO important - hmm). There have been some hacks mentioned in the TS forum that I've tried but they're just that - hacks.
As mentioned elsewhere in this thread, it should be fast, accurate AND reliable (not just the usual "pick any two"). It should have an event-based, multithreaded architecture and access to some sort of scripting language is assumed available. I've grown accustomed to EasyLanguage (OOEL) in particular as I can extend it via DLLs as needed (using whatever language I care to develop in) and I've found I can do a lot more with fewer lines of code in EL. In a perfect world, I'd be able to debug OOEL via VSCode (not just edit), alas, that probably will never happen. My panacea open source platform would translate and compile EasyLanguage/OOEL indicator & strategy code to LLVM and let me run at maximum speed like most retail platforms do today already - someday...
As Mike and others have mentioned, there are a few of us here who decided to bite the bullet and try to roll their own platform as there simply isn't an open source option that I've ever seen that comes remotely close to the commercial retail platforms. If all you're looking to do is relatively simplistic bar-level strategies (MA crossovers, momentum-based, etc.), you probably don't need what I'm looking for which extends beyond that into trying to automate discretionary tick-level price action activity that my eyes can see and process but I've yet to convert 100% to code. Despite having spent years trying literally thousands of bar-level strategies, I've never been able to consistently match what I can do manually using price action alone which is somewhat more difficult to wrap in some algorithmic "secret sauce" (at least for me). Attempts at leveraging machine learning made me realize even that was more about having the right feature sets up front; "indicator soup" is only half (at best) of the data required. so plenty more investigation is still TBD on that front.
Since I also want to be OS-agnostic, I initially thought of seeing what I could do with Python alone and developed Fu510nTrader but ran into performance issues that forced me to stop and re-think architecture. Were Mojo more mature and available for native Windows development (and had support for Qt6), I might have gone that route alas that's probably at least a year+ away from reality. Still, I learned a lot along the way, particularly on UI best practices when writing Qt applications. The platform does do everything on my wish list but has many gaps to fill before ever being able to use for live trading. I decided to tackle the performance issues by porting the whole thing over to Rust last year, which is 100X-1000X faster for sure but here again, developing anything in Rust has a considerable (vertical?) learning curve which took me a year to port and it's still very much "in progress" as a PyQtGraph equivalent really doesn't exist for Rust and Qt's charting API leaves a lot to be desired. Charting alone is a rat's nest of it's own which is making me think I should really just try to integrate with one of the JavaScript-based frameworks like Highcharts or ChartIQ (the former can be used for personal projects).
Which is all just a long-winded way of saying - it's a LOT of work, especially for one person (or even a small group) to attempt alone but having invested several thousand hours in the code I've written over the years, I can see how/why most are reluctant to simply "give it away" and have others potentially profit from one's efforts. I'm not sure really where I stand at this point as I have a few remaining milestones that keep me 150% busy most of the time these days. Packt did reach out to me last year to write a book on mastering EasyLanguage language on TradeStation that I turned down for a few reasons (lack of time be the primary one) though I think we'd all really benefit if there were a FOSS trading platform that literally "did it all" the absolute best way possible and, like Linux, blossomed well beyond what Linus initially started out with.
Thoughts?
-Guy
FWIW, I asked ChatGPT for "an itemized list of all the features in TradeStation 10 (do NOT summarize, but list all of them)" just to see what it would come up with:
1. Advanced Charting Tools
Interactive Charting Interface
Multiple Chart Types (e.g., candlestick, bar, line)
@Fu510n No idea if of value given your experience but fwiw just noticed a reference to a 'Quant Science' on twitter/X. Their handle is " @Quantscience_ " (note the underscore character). Scroll down a bit for a webinar on Jan 15.
FWIW, in fairness to MotiveWave, I haven't used it for over 6+ years as I felt it lacking on several fronts when I tried trading with it (and not being a fan of Java for anything expected to perform well).
I just updated to the latest version a couple days ago and have been pleasantly surprised to find that it actually does tick quite a few items on my list:
Real-time historical tick playback (works pretty well with IQFeed data)
The ability to simply click on a bar to specify a replay starting point (not exactly this but you can specify a HH:MM starting point)
Accurate Renko bars with high/low wicks
Order flow functionality similar to what's provided by Michael Valtos' OrderFlows software built-in
Ability to update indicator parameters "live" and see the effect on the UI immediately (think discretionary "tuning")
Support for Windows, Linux and OSX
It does consume quite a bit of CPU while running but that could be the OFA add-on I also bought back then (2017) so I may just disable that if I don't end up using it to see if that keeps my CPU fan from spooling up (it uses 2X-3X of what TradeStation and/or MultiCharts uses and those both have a lot of charts/indicators on them vs. a couple charts with 2-3 indicators). On a positive note as well, it support TradeStation as a broker while using IQFeed data which is nice.
Having spent a bit of time refamiliarizing myself with the latest MotiveWave version, I was sad to find that the strategy optimizer is still single threaded (vs. pretty much every other platform that has the ability to saturate all cores while optimizing). Not sure why that's the case given it's a Java app, but hopefully something they enhance in future versions (I have the Pro version FWIW).
The ironic thing about all these platforms trying to improve optimization times is that it encourages so many bad habits (primarily over optimization, curve fitting, whatever you want to call it).
Most strategy builders would be better served if these platforms made optimization TOUGHER AND SLOWER.
Thanks for a nice post. Some time ago I started writing my own app suite, initially for some of the reasons that you mention. Apologies for my post not being so nicely structured as yours.
Speed / Charting
When starting out, I was considering whether to go C++ or C#. I opted for C# and I don't regret it. The only upside to using C++ is potentially getting a slight perf increase in some specific parts of the code, but that can always be made up for by buying a better CPU. It's been about 10 years since I last wrote something in C++, so maybe now things are a bit different with C++XY, but at least back in the day, writing in C++ was considerably slower/less comfortable than writing in C#. And I say this while hating every part of garbage collection.
BookMap, JigSaw ... nowadays improve their apps by charting on the GPU using DirectX instead of using CPU. NinjaTrader has been using SharpDX for a long time, so one might be inclined to think that 'fast charting' can't be done on the CPU. I am charting using GDI+ in WinForms and it is FAST. I tried SkiaSharp and other modern frameworks and for the basic lines, rectangles etc there is no measurable perf benefit.
I understand all too well the developer itch to get things perfectly efficient. I took me some time to admit that GDI+ is perfectly fine despite being 20 years old. It took me some time to just let go of the 'core' perf libraries of my backtester being written in CLI/C++ so that I can get the extra 2% perf gain. What I’m trying to say is that the decision of language between Rust, Go, C++, C# .. or whatever other compiled language with high perf support should come from what you can develop your code quickest in, cause the perf differences are going to be very small and unnecessary, unless doing HFT.
Mental side
The hard part of developing something like this is that when you're going alone, the initial enthusiasm wears off in a couple of months, you have nobody to consult with, it can become a grind fast. Doing it in a group could remedy these problems, but getting the other devs is not easy - they have to be deep enough in trading to understand that all of the features are important, they have to be good enough devs, they have to be willing to do this for free ... etc. If the devs knew from the start that after finishing the software, the group effort would continue by developing trading strategies/knowledge together, that could help get people motivated to write the software together. But working on trading strategies together requires that nobody be a freerider, everybody understands that group effort is better for everybody … and that’s yet another requirement on the devs.
Some time ago, I was adding support for trade conditions and darkpools to my app suite. I got to know some devs who were 'deep' into this and we're for example tracking intermarket sweeps for large-cap stocks. Their logic was illogical to me, but when I tried to discuss, I got the cold treatment that "no, this works the way I’m saying because [repeat illogical reasoning here]". So, the endgame of developing things together also requires upfront that all the people have an open mind and are not married to their trading concepts - yet another requirement.
Every time I thought about getting other people to work with me on this, the only solution I found was hiring them. If you think my view of the situation is not right / pessimistic, please tell me because I would be happy to find some positive solution to this.
The main reason I’m writing about this mental side is that you mention open-source and cross-platform. Both of these make the requirements on you, potential team of devs, the scope of the project, used technologies … harder. Given that trading is costly to do right, if somebody is using a Mac, and they want to use your software, THEY should get a PC instead of you having to bother with a solution that can be deployed on a Mac. If the tech stack you use allows for an easy deployment to a Mac, that’s a cherry on top, but in my view should not be a requirement.
Desired functionality
As far as I know, of your wish list NinjaTrader doesn’t have
• Support for Linux/Mac (but maybe I’m wrong)
• Ability to make replay jump by clicking on a bar (but you can jump to a certain time)
• OFA is not built in, but can be purchased or coded
• EasyLanguage
As for backtest speed, TradeStation used to be really slow and would happily crash towards the end of a multi-day backtest – if that is still the case, I understand its lack of appeal. NinjaTrader definitely uses indicator caching. I don’t know how fast NT is in backtesting these days, but having profiled my code extensively in both C#/C++ I can say that there is a perf ceiling you hit pretty quickly, even when everything your strategy uses is precalculated and the critical parts are unsafe/ptr-based. Even if you can somehow outperform NinjaTrader’s backtester performance by a factor of 2x, it is not going to allow for adding another optimized parameter for free, unless its binary.
Im not trying to say you should convert to Ninja, Im trying to say that if it can be bent to meet 95% of your needs, it might be the easiest solution, where the remaining 5% would come at a high cost of developing stuff from scratch.
To end positively, writing my own app suite was definitely exciting and if you’re planning on doing it, go for it. I wish I had had someone to write mine with, as I would probably have avoided some unnecessary coding (like adding support for options) and some dead ends that only cost me a ton of time/money (like anything related to iqfeed mbo data, what a bunch of losers they are). If you can get a coding buddy, that’s gonna help a ton.
EDIT: This was not my best-articulated post, so I will try to sum it up
1/ congrats on deciding to write your own trading apps
2/ you want to make this as easy to do as possible because motivation will run out before you see some applicable results of your work
if I were you, I would cut the nice to have features and didnt think about them until you felt the need to use them
from my point of view, this includes the ability to edit EL unless you're familiar with parsing, tokenizing, compiling ... etc
from my point of view, this also includes support for OSes that you don't use yourself. Supporting them increases work for you while the reward for your is questionable - the good feeling that you have a killer app that supports Mac will I think not keep you going
3/ don't focus on runtime performance, most compiled languages that have some perf part in them (e.g. Unsafe and Span in C#), are pretty much in the same perf league and as a user you will not notice a +/- 5% runtime difference. Choose a language that you're cool with as this will likely be a 1-2 year project and complicating it by learning Rust first is going to make it a 2-3 year project just to make it 5% faster in a good case scenario
At the beginning, I wanted to have all the features coded perfectly. When an idea for a new feature came, I couldn't get it out of my head until I coded it. Now, some two years and 80KLOC later, I am wondering why I was coding equity options split adjustment, when I wanted to predominantly focus on futures I want you to avoid the mistakes I did. Cheers!
I want you to avoid the mistakes I did. Cheers!