I have created a application for storing tick data to a database. The app is a server that a client can connect to and perform reads/writes of tick data. I call it SimpleTickServer. I have provided a client implementation in DLL form as well. In order to support backfilling I modified GomFileConverter slightly. I also had to modify the GomRecorderIndicator slightly to abstract the storage a little more. Gom files have price and tick type, but my app needs price, bid, and ask. The changes are minor, and I hope Gomi will roll them into his "standard" release.
The DB implementation is Berkeley DB, it is NOT an SQL database. It can only be queried programatically. It stores all ticks for a unique time stamp and symbol (to second resolution) together. Requests to write a symbol/timestamp replace all existing data with the new data. Read requests are done by positioning the with a setup request to a symbol/timestamp and then issuing read next requests. Each read request returns all the ticks for the next second and advances the position so subsequents reads will get the next timestamp for that symbol. The client implementation I provided handles all read and write buffering including figuring out when there is no more data to read.
I have done a moderate amount of testing with the modified file converter, but there are likely problems remaining. One issue ... I haven't been able to actually get it to work in ninja trader! With an updated GomRecorderIndicator (see the attach gomfileconverter VS project for this file) set to SimpleTickServer I never see an incoming client request at the server end. Is it possible to run NT indicators in a debugger?
I'm attaching everything I have here, in the hopes it will be useful to some of you, and maybe I can get help making it work in NT. Here are the attached files:
GomFileRecorder_SimpleTickServer_VS2010Project.zip: A visual studio project to rebuild the modified GomFileConverter.
GomFileRecorder_SimpleTickServer.zip: This is a modified recorder for reading data from and backfilling the database. I added a symbol field to the GUI, you must specify this when backfilling data, and it needs to be the Symbol NT uses. Example ES ##-##. The source file MUST have full bid and ask data (IRT, Qcollector .. etc), I modified the QCollectorIQ to work with the files QCollectorIQ produces for me, they are ',' seperated not \t. I also added QCollectorIQ_OHLC, it parses OHCL data for backfill, price=bid=ask at each price of a bar with volume distributed evenly. I use this for testing and I want to build long term Volume Profiles. I plan to create a full tick (price/bid/ask/volume) file output option, but haven't yet. The SimpleTickServer*.dll files are needed by the GomRecorderIndicator_SimpleTickServer.cs file in order to build.
BerkelyDB.zip: 32 bit build of the BerkelyDB 5.1.19 version. These must reside in your PATH.
SimpleTickServer_VS2010Project: Source and Visual studio project files for build the application and client DLLs. The structure kind of sucks ... but this is my first time really using MS visual studio and I didn't want to waste a lot of time creating a perfect project layout.
SimpleTickServer: The application. For now it runs hard coded to localhost:3861 the client knows this because it is to, so it shouldn't have trouble finding it, I plan to make it configurable in time. The only command accepted at the prompt is "exit", which flushes everything and quits. It stores the data in your User directory unless you specify a SIMPLETICKSERVERPATH environment variable with the path you'd like it to use. It doesn't trim it's own log files right now, so they can grow pretty big. You shouldn't have trouble deleting the older ones when the server is shutdown. I plan to add configurability for thinks like cache sizes (256MB hardcoded now) and commands at the prompt for getting stats like the cache hit rate. Performance so far seems ok, it's much faster to read than to write.
Right …