Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I don't know how IQfeeds plaftorm works and how it looks like but why does IQFeed's own historical download platform not download full 120 days? Can you explain that? So no AB, Sierra chart etc involved. And why does Sierra Chart mention a limit during market hours? That's kind of a contradiction.
Can you help answer these questions from other members on NexusFi?
Look, I think we can safely stop corresponding. I was hoping it was user error when I created the thread. It is not user error, but AmiBroker error. There is no need to have this conversation both in the thread, and with Tomasz at AmiBroker.
To briefly answer your question, which I have already told AmiBroker, the issue is with requesting the data all at once. You should not do this, but rather break it into chunks. If AmiBroker had tested the 120 days of tick data backfill, they should have noticed there was a problem with their implementation prior to me bringing it to their attention.
While I was securing the proof that IQFeed even had 120 days of tick data, because Tomasz initially told me I was "dreaming in fantasy land" if I thought they did, I also talked to IQFeed about the problem and the first thing they commented on was that most likely the issue was AmiBroker not splitting the request into chunks. It is a common thing.
As for market hours, I do not know what you are implying. It is well known IQFeed has a market hours and after market hours policy. I am not confused by this and it has nothing to do with the problem.
You'd have to ask Tomasz. It is his figure.
This is my final reply in this thread until there is progress in one way or another from AmiBroker.
I only mentioned what has been tried by others. For example I have tried with another user who uses IQfeed on 5 minute timeframe to download tick bars and failed to download complete 120 days or rather the complete 10 million maximum tick bars been set. Also that other befriended user has sent a request to IQfeed today and they replied
And if IQFeeds own application also doesn't download full 120 days without any other software involved then there indeed seems to be a problem with their download. Again I can't confirm the last one as I haven't used it myself but it has been told to me today because I was interested in that issue myself.
I'm not sure why and how I should do this? AB is not open source and therefore I don't have access to AB's source code.
After some more email exchanges with Tomasz, the conclusion is that he has reproduced my issue but says that no one else has ever asked for this -- so he deems it is low priority and says it will take "months at least" for him to fix it.
I cannot help but feel like the fix should not take months. I think he is mad at me personally...
Anyway, in his defense he offered a refund of AmiBroker over the problem. I told him I've had it for only 1 day and would really like more time with it, 120 days of IQFeed backfill is not the only thing that is important to me, so if it does other things well then there would be no need for a refund.
I am still hopeful that it is as optimized and powerful as I have read. More time will tell.
Again, the IQFeed policy limit is well known and has nothing at all to do with my issue.
Yes, I spoke to Jay at IQFeed and he confirmed that Tomasz found a bug and IQFeed will patch it shortly. But I think you are missing the point. My understanding of the good practices is to retrieve the backfill in chunks instead of making a single request. AmiBroker seems to make a single request, resulting in timeouts, overflow errors, or similar. All other charting programs do not have this problem, so from my perspective it seems like IQFeed has no issue filling the backfill requests with other programs and AmiBroker should be able to do the same.
In the end, he told me a "patch" would be coming -- in several months.
There is no need to continue going back and forth.
I guess you need the (long) history of 120 days of tickdata for backtesting purposes.
So you could build an offline database and import tickdata that is downloaded with QCollector. Use an offline database as you should not mix online (realtime) datafeed with imported data.
The import process can be automated on both sides: Qcollector and Amibroker provide convenient ways to do that.
In Amibroker use this approach:
Tools > Customize... [Tab:Tools] > [Button: New]
Command: #import
Arguments: your path to local stored tickfile
Initial directory: your path to local format file.
See "Update US Symbol list and category" as a starting point. Of course this example only loads symbols and categories, but you should get the point.
Once done, you can import data with a single mouse-click.
But be warned: in online mode (with real data) use a separate database. Never mix manual backfills with realtime data.
BTW: You can start multiple instances of Amibroker simultaneously.
Do you know how those 40 bytes are generated ?
I think
1 double has 8 bytes, right ?
So 5 values in a bar structure OHCLV but Date is missing
Therefore a bar would be structured in the following way.
4 bytes for a int for the Volume maybe also a short 2 bytes would do
4 double with 8 bytes
and the rest for the Date Time
This leads to the conclusion that for storing tickdata
a tick structure of 8 bytes (volume and Date Time) and 8 bytes for Price is needed
2 million ticks/day * 120days * 16 bytes =3.84 GB
But I am not sure if that is needed in the RAM because there is always only a limited lookback period.
Even if I will display the complete data by shrinking my timescale in an chart, i guess that the chart application will reduce the needed data in an intelligent way. Because the eye cannot see every tick and the indicator calculation is maybe only done on slices. I think that is the way a good program should be reduce data.
Maybe that could be the reason why it would take that long for Tomasz to implement that feature.
Because maybe Amibroker has no strategy to deal in the above discribed way with data.
Yes the 120 days of IQFeed was just a fast simple test, I wasn't expecting to hit a brick wall. 120 days of tick data is not useful by itself, it's just useful for backfill. I have my main MySQL databases with years and years of tick data for many instruments, which I would need to import to do any useful testing.
I have not yet started that process, and will likely make a separate thread to discuss it.
One thing in your reply concerns me and caught my attention however --- you said "never mix realtime with historical", why is that? This is a concern, because my typical behavior is to import the data from my database into the platform, then use the platform to backfill and track data going forward -- both realtime and backfill as needed. Can you go into more detail on this?
Cool, I did not know you could run multiple concurrent instances of AmiBroker so that is good. I also had meant to do some research about what the licensing allows --- is it a single user license, but can be deployed on multiple pieces of hardware (ie: my development workstation + my production server) concurrently, as a single user? Or would it require multiple licenses. Need to check on that still.