https://www.cmegroup.com/confluence/display/EPICSANDBOX/Matching+Algorithms

combined with

https://www.cmegroup.com/globex/files/globex-product-reference-sheet.xls

well let you see exactly which product uses which matching Algo

iantg 

I don't typically give too much away about my personal system, but in this case I will share because this method is widely known and widely used by all HFT systems. I am punting this out there also because I have a few questions, and I am hoping that someone out there has an answer to my question at the end, we will see... But here is the concept in a Nutshell. When flat, you submit both buy and sell limit orders 4 levels above and 4 levels below the current level, so you have 8 orders resting at one time. My goal is simple, I just want to get one of these 8 orders filled within the first two sequences of action when the market moves to that level. A typical sequence once the price moves up looks like this: Seq 1. Resting Bids gets served first. Usually 50% to 75% of the resting amounts get served. Seq 2. Resting Asks get served next. Usually 50% to 75% of the resting amounts get served. Seq 3. Resting Bids get served again, the remaining 25% to 50% are worked on here, but not completely exhausted. Seq 4. Resting Asks get served again, the remaining 25% to 50% are worked on here, but not completely exhausted. Seq 5. The final Bids that were resting that did not yet get filled + any new bids that entered are being served Seq 6. The final Asks that were resting that did not yet get filled + any new bids that entered are being served Etc..... With the ES, 55% to 65% of all sequences that start with price up level change, and start by serving the Bid queue first, will end up depleting the bids before the asks are depleted, so the next level will be down. Vice Versa for levels that move down and start with the asks first. This is due to the fact that by sequence 3 you will have 2 attempts to clear the side that went first and only one attempt on the side that went second. By level 5 this will be 3 to 2 and by level 7 this will be 5 to 4. So this is why there seems to be so much noise in price movement in the ES, but typically one level that clears is the signal, and the next opposite level that clears is the noise. Determining which level is noise and which level is the signal can come as a function the count of levels cleared prior to the change, or duration of time on a level, but this is beyond the scope of this topic. (I may cover this in depth in later posts if there is adequate interest).

I punted this topic out here a couple days ago, and it has a lot of interesting points to it. I can tell you that the mechanics of how the bids and asks fill typically follow what I have described above. I only know this, because I have observed this by extracting millisecond level data (Millions of rows) and analyzing them. I have some theories on this behavior, but honestly I don't quite understand the exact drivers of the following: I know how it kind of works but I would love to know more about it.

At any point in time the Fill engine has two options.

1. Fill all the resting volumes on a single side and drive the price up or down to the next level accordingly.
2. Fill only a portion of the volumes on one side, and the stop and switch and fill from the other side keeping the level in tact.

I figure there are a few possible reasons for why it moves the way that it does, but none of these quite make sense. Here are a few theories that I think would be easy to debunk:

Theory A: The FIFO may apply to both sides evenly. So it uses time stamps on both the bids and asks indiscriminately. Whichever got there first gets worked on, this is why it flips the last all the time.

Obvious Issues: If this was the case the last would jump back and forth hundreds of times on each level, but I only typically observe 10 or so moves back and forth and each with a a significantly clearing of the level, so this doesn't add up.

Theory B: There is a special ratio that the fill engine uses to hit a predetermined portion on pass 1, pass 2, pass 3, etc. This is all ratio driven and based on the volumes that are out there X amount will be cleared before switching sides.

Obvious Issue: If such a targeted method was in place the patterns would be more consistent. I don't observe this.

Theory C: This could be pricing related in terms of average price movements: If a certain portion of bids or asks clear then the weighted average of the spread relative to recently cleared volumes drives the last price up or down. When it gets passed the median threshold it changes which side gets served. This is all just a function of pricing.

Obvious Issue: The line that divides this would get crossed every few trades, and the logic to find a price related balancing point would be like sprinkling salt on two sides of an gram scale.. It would just swing back and forth with every few flakes of salt you added.


So for bonus points, does anyone have any clue exactly the reasoning for why the fill switches sides between the bid and ask, as in what is the key trigger for this event. I have tried to look this up but I have never found anything on this topic anywhere. I can observe the data and figure up statistics about it, but I can't find any information governing the logic in the mechanics. I would appreciate any insight anyone has on this topic. And if anyone wants any of my research on this, just let me know.


Thanks!

Ian

My guess is that it might be something like this:

Price is moved by market orders (also called aggressors) as we know. Market orders on both sides are also put into one FIFO queue and serviced accordingly. So as an example for a fictional instrument that uses the FIFO algorithm:

Best Bid : $105.00 - 100 limit orders resting
Best Offer : $106.00 - 250 limit orders resting

Market orders arrive in the following order:

Sell Aggressor : 3 contracts
Sell Aggressor : 5 contracts
Buy Aggressor : 9 contracts
Buy Aggressor : 10 contracts
Buy Aggressor : 6 contracts
Sell Aggressor : 1 contract
....

The way this will play out in the matching engine is: Bid price->Bid Price->Offer Price->Offer Price->Offer Price->Bid Price.

This one queue of market orders will be serviced on a first in first out basis until all the resting limit orders on a level are exhausted and then move on to the next level.

Once again, this is my guess.

What do you think?

Popsicle 

My guess is that it might be something like this: Price is moved by market orders (also called aggressors) as we know. Market orders on both sides are also put into one FIFO queue and serviced accordingly. So as an example for a fictional instrument that uses the FIFO algorithm: Best Bid : $105.00 - 100 limit orders resting Best Offer : $106.00 - 250 limit orders resting Market orders arrive in the following order: Sell Aggressor : 3 contracts Sell Aggressor : 5 contracts Buy Aggressor : 9 contracts Buy Aggressor : 10 contracts Buy Aggressor : 6 contracts Sell Aggressor : 1 contract .... The way this will play out in the matching engine is: Bid price->Bid Price->Offer Price->Offer Price->Offer Price->Bid Price. This one queue of market orders will be serviced on a first in first out basis until all the resting limit orders on a level are exhausted and then move on to the next level. Once again, this is my guess. What do you think?

Thanks for taking a swing at this!

You know this was another scenario that I thought about but I left it off my list of guesses because market orders are typically small sized. While everyone including big players use them for stop losses, this is the most common order type used for entries by the small retailers.... So this would imply that you would see tons and tons of 1-3 contract market orders. If the fill logic was to just flip long and short entries from retail traders against the resting bid and ask limit orders, then I would expect to see the market flash like a strobe light alternating the last price almost every millisecond. So every level that clears would have thousands of flips between the bid and ask. What I typically observe is 3- 10 flips of the last price as the bid volumes and ask volumes are being served on each level. So I would be surprised if this alone was it. But it is possible that the timing of the market orders has something to do with it overall. I just don't think time alone is the factor for the reason I outlined.

I would love to know the answer to this query though. I think that a ton of successful trading techniques could be derived from uncovering this.

Ian

With a FIFO engine what @Popsicle said is I believe correct.

Remember the matching engine is only processing orders when they come in. Without new/change orders nothing happens. At time it seems like you are implying the matching engine decides whether to fill the bid or ask next. That is decided by whether the next aggressor order is a buy or a sell and nothing to do with the engine itself.

Here's another theory. What percentage of the ES market do you think is HFT or other automated trading systems? 50%? If so half of all those trades are those systems taking position, exiting positions, re-positioning themselves, etc etc.

SMCJB 

With a FIFO engine what @Popsicle said is I believe correct. Remember the matching engine is only processing orders when they come in. Without new/change orders nothing happens. At time it seems like you are implying the matching engine decides whether to fill the bid or ask next. That is decided by whether the next aggressor order is a buy or a sell and nothing to do with the engine itself.

I am inclined to say that I think this theory makes the most sense and I would believe this to be the case except the data I have collected doesn't quite line up.

The enclosed spreadsheet captures every change in level one position when the last price flips between the bid and ask. I have this sequenced up to 7 sequences. (beyond this was unnecessary because the level usually broke prior to this.) Here is how the sequence works:

If the level moves up:
Seq 1 = Last = Bid and bid volumes are cleared
Seq 2 = Last = Ask and ask volumes are cleared
Seq 3 = Last = Bid and bid volumes are cleared
etc.

If the level moves down:
Seq 1 = Last = Ask and ask volumes are cleared
Seq 2 = Last = Bid and bid volumes are cleared
Seq 3 = Last = Ask and ask volumes are clared
etc.

There are a few exceptions, but like 90% + of the time if the move is down, the ask volumes go first, and if the move is up the asks go first. Considering the ES typically has a few hundred resting bids and asks on both sides and considering that most market orders are smallish < 10 contracts, to take down an entire level I would expect to see hundreds of flips between bid and ask if this was just FIFO on market orders.

I have my code built to run an update on the OnMarketData event handler every time the last price flips, and I have it sequenced so it always knows the count up to level 7. Everything is color coded so it's pretty easy to follow. This is just a cross section of an hour or so from the ES last month, but this is how I am seeing the market move and levels cleared.

I would love to understand the mechanics of this more, and it is possible that market orders are driving this (Flipping of the last) but I just don't quite get how.

Anyway, I am punting this out there with some actual data, to try to spark a lively conversation. Hopefully we can crack this.

Thanks to everyone pitching in throwing in ideas!

Ian

before do the backtest you have to do the double check that the ticks who you have are correct,
you can try with tickstory

Are you sure your data is complete?
Row 2 is 2654.75/2655 341x155
Row 3 is 2654.75/2655 154x322 (ie bid volume down 187, ask volume up 167)
Obviously one order can not both decrease the bid volume and increase the ask volume without changing price levels.
I suspect your not getting the raw tick data but something where updates are conjugated before being distributed. If so, then what appears to be one update to you, could be made up of dozens of updates.

SMCJB 

Are you sure your data is complete? Row 2 is 2654.75/2655 341x155 Row 3 is 2654.75/2655 154x322 (ie bid volume down 187, ask volume up 167) Obviously one order can not both decrease the bid volume and increase the ask volume without changing price levels. I suspect your not getting the raw tick data but something where updates are conjugated before being distributed. If so, then what appears to be one update to you, could be made up of dozens of updates.

I have thought this also. I have ran this type of script on 1 tick, 1 second and the OnMarketData event handler in NT. https://ninjatrader.com/support/helpGuides/nt8/en-us/?onmarketdata.htm And tried to get it as granular as possible.
The issue that I keep hitting is that the resting volumes seem to have obvious gaps, even though the time stamps seem to have millisecond precision. The data is supposed to be unfiltered, it's just the standard replay data from Ninjatrader / Kinetic. I might record some live data from my Continuum data feed and play it back and see if there is any more granularity there.

The only possible explanation I can think of is that 25% to 50% of all the volumes we see are being canceled before they actually execute. This might be the case on the ES with all the HFT. So this might explain why level changes occur when the last published volumes are north of 0 by a large margin. Or... I just have serious gaps in my data somehow.