I'm not sure if this is the correct place to post, so please advise if another thread is more suitable.
After seeing Big Mike post about his trading platform, I've been inspired to improve my own. After analyzing my strategy development over the past few months, it's become clear that my largest consistent time drain is getting data to a workable format for strategy testing/development. My goal is to create a database that is compatible across windows / unix and can output its data to any program/language so that I can use it in the future regardless of language.
Programming Languages to use:
- Matlab
- Java
- SQL
Data feed:
- IQFeed main data source
- Other longer-interval data sources (Reuters, FRED, etc...)
My current Database:
- Tick Database:
- Data: Tick data based - L1 (Bid/Ask/Last), L2 / Market Depth
- FileType: Flat File database
- FileFormat: Custom binary format file separated by symbol, date, and feed type (L1, L2/Market depth)
New features to include - longer data intervals (best database type?):
- Bars - Sec / Minute / Daily bars (Currently calculated on demand in backtesting)
- Calculated fields - VWAP, Avg vol, avg vol for time of day, etc...
- Fundamental data - News Events/time, VWAP, etc....
Any advice on the longer-interval database type would be much appreciated. I saw mike uses a MySQL database, I've got some SQL experience so I'm leaning toward that but would love to hear any comments on why/why not use it.
Also, I'm a noob when it comes to programming. My background is mechanical engineering and I've really only picked up programming in java over the past year. There are many things I don't know I'm doing wrong, i.e. I never use unit tests because it has never been at the top of my priority list, so if you see me making any noob mistakes please let me know.