I've written a Tic datastore using LZ4 Compression and Unsafe serialization that stores Trades, TopOfBook Updates (Bid/Ask), Depth Updates, and News Events.
My goals were the following:
- Ensure no duplicates - When adding files to the database ensures I don't accidentally double up on the same events.
- High Stream Rate - Use LZ4 and Unsafe but allow for changes in the future if another library performs better
- Asynchronous - Data stream would be too large to dump into memory, so force asynchronous
This will be used for both backtesting and data analysis. All of my data bars will be based off this information. After completing the database here is the performance that I'm seeing