Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Flat text file-based storage with some clever file naming convention and maybe a directory hierarchy to split contracts by markets and types (i.e. futures from stocks from options, etc...) is a more scalable solution than a traditional DB. Most of the time when you work with historical data (time series) - you don't need to query data, or update /delete records, you just load it up as a sequence of data points, and reading files line-by-line works great. Most languages have support for streaming, so you don't need to load all data into memory at once..
So there is often no need for a SQL-based (or even a non-SQL based) database. Market price data, once obtained and cleaned, is static. You can zip it to save more space, and it's easy to archive. In addition, no DB means one less system to setup, maintain, backup, worry about fail-over, etc... Flat files do not consume CPU power or memory until they are used, unlike a DB engine. You can also easily load comma- or tab- separated files in Excel, or any other tool of your choice.
By the way found a really great tool for getting stocks/futures/options/Forex historical data as csv text files from Interactive Brokers, highly recommend it. You can forget about pacing violation or request/response size limits , it splits large data requests into many smaller once automatically behind the scenes: