Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I punted this topic out here a couple days ago, and it has a lot of interesting points to it. I can tell you that the mechanics of how the bids and asks fill typically follow what I have described above. I only know this, because I have observed this by extracting millisecond level data (Millions of rows) and analyzing them. I have some theories on this behavior, but honestly I don't quite understand the exact drivers of the following: I know how it kind of works but I would love to know more about it.
At any point in time the Fill engine has two options.
1. Fill all the resting volumes on a single side and drive the price up or down to the next level accordingly.
2. Fill only a portion of the volumes on one side, and the stop and switch and fill from the other side keeping the level in tact.
I figure there are a few possible reasons for why it moves the way that it does, but none of these quite make sense. Here are a few theories that I think would be easy to debunk:
Theory A: The FIFO may apply to both sides evenly. So it uses time stamps on both the bids and asks indiscriminately. Whichever got there first gets worked on, this is why it flips the last all the time.
Obvious Issues: If this was the case the last would jump back and forth hundreds of times on each level, but I only typically observe 10 or so moves back and forth and each with a a significantly clearing of the level, so this doesn't add up.
Theory B: There is a special ratio that the fill engine uses to hit a predetermined portion on pass 1, pass 2, pass 3, etc. This is all ratio driven and based on the volumes that are out there X amount will be cleared before switching sides.
Obvious Issue: If such a targeted method was in place the patterns would be more consistent. I don't observe this.
Theory C: This could be pricing related in terms of average price movements: If a certain portion of bids or asks clear then the weighted average of the spread relative to recently cleared volumes drives the last price up or down. When it gets passed the median threshold it changes which side gets served. This is all just a function of pricing.
Obvious Issue: The line that divides this would get crossed every few trades, and the logic to find a price related balancing point would be like sprinkling salt on two sides of an gram scale.. It would just swing back and forth with every few flakes of salt you added.
So for bonus points, does anyone have any clue exactly the reasoning for why the fill switches sides between the bid and ask, as in what is the key trigger for this event. I have tried to look this up but I have never found anything on this topic anywhere. I can observe the data and figure up statistics about it, but I can't find any information governing the logic in the mechanics. I would appreciate any insight anyone has on this topic. And if anyone wants any of my research on this, just let me know.
Price is moved by market orders (also called aggressors) as we know. Market orders on both sides are also put into one FIFO queue and serviced accordingly. So as an example for a fictional instrument that uses the FIFO algorithm:
Best Bid : $105.00 - 100 limit orders resting
Best Offer : $106.00 - 250 limit orders resting
The way this will play out in the matching engine is: Bid price->Bid Price->Offer Price->Offer Price->Offer Price->Bid Price.
This one queue of market orders will be serviced on a first in first out basis until all the resting limit orders on a level are exhausted and then move on to the next level.
You know this was another scenario that I thought about but I left it off my list of guesses because market orders are typically small sized. While everyone including big players use them for stop losses, this is the most common order type used for entries by the small retailers.... So this would imply that you would see tons and tons of 1-3 contract market orders. If the fill logic was to just flip long and short entries from retail traders against the resting bid and ask limit orders, then I would expect to see the market flash like a strobe light alternating the last price almost every millisecond. So every level that clears would have thousands of flips between the bid and ask. What I typically observe is 3- 10 flips of the last price as the bid volumes and ask volumes are being served on each level. So I would be surprised if this alone was it. But it is possible that the timing of the market orders has something to do with it overall. I just don't think time alone is the factor for the reason I outlined.
I would love to know the answer to this query though. I think that a ton of successful trading techniques could be derived from uncovering this.
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals, U308 and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,059 since Dec 2013
Thanks Given: 4,410
Thanks Received: 10,226
With a FIFO engine what @Popsicle said is I believe correct.
Remember the matching engine is only processing orders when they come in. Without new/change orders nothing happens. At time it seems like you are implying the matching engine decides whether to fill the bid or ask next. That is decided by whether the next aggressor order is a buy or a sell and nothing to do with the engine itself.
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals, U308 and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,059 since Dec 2013
Thanks Given: 4,410
Thanks Received: 10,226
Here's another theory. What percentage of the ES market do you think is HFT or other automated trading systems? 50%? If so half of all those trades are those systems taking position, exiting positions, re-positioning themselves, etc etc.
I am inclined to say that I think this theory makes the most sense and I would believe this to be the case except the data I have collected doesn't quite line up.
The enclosed spreadsheet captures every change in level one position when the last price flips between the bid and ask. I have this sequenced up to 7 sequences. (beyond this was unnecessary because the level usually broke prior to this.) Here is how the sequence works:
If the level moves up:
Seq 1 = Last = Bid and bid volumes are cleared
Seq 2 = Last = Ask and ask volumes are cleared
Seq 3 = Last = Bid and bid volumes are cleared
etc.
If the level moves down:
Seq 1 = Last = Ask and ask volumes are cleared
Seq 2 = Last = Bid and bid volumes are cleared
Seq 3 = Last = Ask and ask volumes are clared
etc.
There are a few exceptions, but like 90% + of the time if the move is down, the ask volumes go first, and if the move is up the asks go first. Considering the ES typically has a few hundred resting bids and asks on both sides and considering that most market orders are smallish < 10 contracts, to take down an entire level I would expect to see hundreds of flips between bid and ask if this was just FIFO on market orders.
I have my code built to run an update on the OnMarketData event handler every time the last price flips, and I have it sequenced so it always knows the count up to level 7. Everything is color coded so it's pretty easy to follow. This is just a cross section of an hour or so from the ES last month, but this is how I am seeing the market move and levels cleared.
I would love to understand the mechanics of this more, and it is possible that market orders are driving this (Flipping of the last) but I just don't quite get how.
Anyway, I am punting this out there with some actual data, to try to spark a lively conversation. Hopefully we can crack this.
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals, U308 and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,059 since Dec 2013
Thanks Given: 4,410
Thanks Received: 10,226
Are you sure your data is complete?
Row 2 is 2654.75/2655 341x155
Row 3 is 2654.75/2655 154x322 (ie bid volume down 187, ask volume up 167)
Obviously one order can not both decrease the bid volume and increase the ask volume without changing price levels.
I suspect your not getting the raw tick data but something where updates are conjugated before being distributed. If so, then what appears to be one update to you, could be made up of dozens of updates.
I have thought this also. I have ran this type of script on 1 tick, 1 second and the OnMarketData event handler in NT. https://ninjatrader.com/support/helpGuides/nt8/en-us/?onmarketdata.htm And tried to get it as granular as possible.
The issue that I keep hitting is that the resting volumes seem to have obvious gaps, even though the time stamps seem to have millisecond precision. The data is supposed to be unfiltered, it's just the standard replay data from Ninjatrader / Kinetic. I might record some live data from my Continuum data feed and play it back and see if there is any more granularity there.
The only possible explanation I can think of is that 25% to 50% of all the volumes we see are being canceled before they actually execute. This might be the case on the ES with all the HFT. So this might explain why level changes occur when the last published volumes are north of 0 by a large margin. Or... I just have serious gaps in my data somehow.