Hi everyone,

I am looking to source historical order book depth data for Index Futures (CME ES / NQ) to backtest an order flow model in Python.

Because I am testing on a cloud VM, I cannot parse raw tick-level MBO/PCAP files. I specifically need PRE-AGGREGATED interval snapshots.

Specifications needed:
• Assets: CME Index Futures (ES, NQ, or MNQ)
• Data Structure: 1-minute or 5-minute binned snapshots (Top 5 to 10 Depth Levels: Bid/Ask prices & sizes, Delta/Imbalance)
• Timeframe: Last 6 to 12 months
• Preferred Formats: Parquet, CSV, or SQLite

If you capture/aggregate Rithmic, Databento, or Sierra Chart depth locally and are willing to share or sell a copy of your dataset, please DM me with:
1. Covered date range & symbol
2. Column schema / depth levels included
3. Your expected compensation / price

Thanks!

I think that if you could upload one sample file, showing how exactly you want the output data to be, which particular columns etc. then that would make it much easier for the forum members. You are talking about the BO/PCAP files and yet you want the snapshots to be in 1 min, 5 min timeframes ? Usually this type of data is consumed in the form of tick data and at the most snapshot of 1 second etc. but the 5 minute timeframe is just too big for this type of data, or perhaps I might be missing something. Uploading sample data in the exact format would clarify it.

Many members here do collect such MBO data and someone might help you out for sure. All the best.

4autotrading 

I think that if you could upload one sample file, showing how exactly you want the output data to be, which particular columns etc. then that would make it much easier for the forum members. You are talking about the BO/PCAP files and yet you want the snapshots to be in 1 min, 5 min timeframes ? Usually this type of data is consumed in the form of tick data and at the most snapshot of 1 second etc. but the 5 minute timeframe is just too big for this type of data, or perhaps I might be missing something. Uploading sample data in the exact format would clarify it. Many members here do collect such MBO data and someone might help you out for sure. All the best.

Thanks for reply.
I can accept — basically any format you collect in:
- Raw CME MDP3 (Globex) PCAP captures, with or without your decoder
- Databento MBO/MBP exports (.dbn, .dbn.zst, CSV/parquet dumps)
- Rithmic R|API / R|Replay captures or exports
- TT FIX drop copies / TT exports · CQG FIX drop copies
- dxFeed / iQFeed depth exports
- Broker FIX drop-copy logs (new/replace/cancel/execution reports)
- Sierra Chart / NinjaTrader DOM or depth archives (incl. Market Replay data)
- Any CSV/Parquet/SQLite dump with ts + side + price + size + action (+ order id)
- L2 top-N snapshot streams at 100ms–1s if you don't keep full MBO
- Even raw packet captures if you share a sample.
As I'm still in the backtesting stage,databento prices are above my budget.

4autotrading 

I think that if you could upload one sample file, showing how exactly you want the output data to be, which particular columns etc. then that would make it much easier for the forum members. You are talking about the BO/PCAP files and yet you want the snapshots to be in 1 min, 5 min timeframes ? Usually this type of data is consumed in the form of tick data and at the most snapshot of 1 second etc. but the 5 minute timeframe is just too big for this type of data, or perhaps I might be missing something. Uploading sample data in the exact format would clarify it. Many members here do collect such MBO data and someone might help you out for sure. All the best.

Thanks — that's a fair point, and I think there's a small misunderstanding about the timeframes that I should clear up: we do not want 1-min/5-min bars, and we're not asking anyone to produce them. We already hold aggregated bars for our backtest engine. What we're looking for is the raw event stream (every add/modify/delete/execute on the book), because testing resting-order/absorption hypotheses requires per-order events — no bar or snapshot frequency can represent that. We'll do all aggregation ourselves, so we can accept the data in whatever native format the collector holds (MDP3 PCAP, Databento-style MBO, Rithmic/R|Replay, TT/CQG FIX drop copies, or plain CSV/SQLite dumps of events).

To make it concrete, I've attached a small synthetic example of the event layout we'd like to receive (illustrative only — not real data, and not the only format we accept):

- ts_ns_utc, symbol, side, price, size, action, order_id, ref_order_id, type
- side = the resting order's side; action = A(add) / M(modify) / C(cancel) / D(delete) / R(reduce-partial) / E(execute-full); ref_order_id links executions to their parent order

That's essentially CME MDP3 / Databento MBO semantics. If anyone holds such archives for NQ/ES/MNQ (or any futures), even short periods, I'd be glad to validate a sample and pay a fair price — longer continuous histories (weeks→months) are especially valuable to us. Many thanks to everyone who's replied.

probit 

What we're looking for is the raw event stream

@probit not sure if we're talking about the same thing here, but we do have years of NinjaTrader Replay data available for ES (which you're after) but not NQ, in our Downloads section.

To clarify, Replay data for NinjaTrader can be used with "tick replay" on, which "will ensure that the market data (bid/ask/last) that went into building a bar is loaded in the exact sequence of market data events. This guarantees that your indicators and strategies are historically calculated tick-per-tick exactly as they would have been if the indicator/strategy was running live during a period."

This paragraph quoted is from this link: https://ninjatrader-live.ninjatrader.com/support/helpguides/nt8/tick_replay.htm

Is this at all of interest?

probit 

Thanks — that's a fair point, and I think there's a small misunderstanding about the timeframes that I should clear up: we do not want 1-min/5-min bars, and we're not asking anyone to produce them. We already hold aggregated bars for our backtest engine. What we're looking for is the raw event stream (every add/modify/delete/execute on the book), because testing resting-order/absorption hypotheses requires per-order events — no bar or snapshot frequency can represent that. We'll do all aggregation ourselves, so we can accept the data in whatever native format the collector holds (MDP3 PCAP, Databento-style MBO, Rithmic/R|Replay, TT/CQG FIX drop copies, or plain CSV/SQLite dumps of events). To make it concrete, I've attached a small synthetic example of the event layout we'd like to receive (illustrative only — not real data, and not the only format we accept): - ts_ns_utc, symbol, side, price, size, action, order_id, ref_order_id, type - side = the resting order's side; action = A(add) / M(modify) / C(cancel) / D(delete) / R(reduce-partial) / E(execute-full); ref_order_id links executions to their parent order That's essentially CME MDP3 / Databento MBO semantics. If anyone holds such archives for NQ/ES/MNQ (or any futures), even short periods, I'd be glad to validate a sample and pay a fair price — longer continuous histories (weeks→months) are especially valuable to us. Many thanks to everyone who's replied.

Thanks for your explanations @probit , now it's totally clear, what you are looking for.

Please do check the data that I have sent you, it's exactly of this type only, as it's pure MBO and contains every orderbook event. Unfortunately it doesn't have the exact symbols that you are looking for.

Maybe someone else would chime in with that exact data here.

xplorer 

@probit not sure if we're talking about the same thing here, but we do have years of NinjaTrader Replay data available for ES (which you're after) but not NQ, in our Downloads section. To clarify, Replay data for NinjaTrader can be used with "tick replay" on, which "will ensure that the market data (bid/ask/last) that went into building a bar is loaded in the exact sequence of market data events. This guarantees that your indicators and strategies are historically calculated tick-per-tick exactly as they would have been if the indicator/strategy was running live during a period." This paragraph quoted is from this link: https://ninjatrader-live.ninjatrader.com/support/helpguides/nt8/tick_replay.htm Is this at all of interest?

@xplorer, thanks for sharing the information regarding the Ninjatrader replay data that we have in download’s section, but I am afraid that probit is not looking for that data. What he really wants is MBO. Here is a short explanation for -

THE THREE LEVELS OF MARKET DATA
(Using Databento's terminology here — databento.com/microstructure.)

- Level 1 (L1) — Top of book: Just the best bid, best offer, and the last trade. One price, one size, on each side. This is the single quote you see almost everywhere.

- Level 2 (L2) — Market depth, also called MBP (Market-by-Price): Adds visibility into several price levels beyond the top (e.g., the next 10 prices on each side). But it's aggregated, at each price you only see the TOTAL size resting there, pooled across however many separate orders happen to sit at that price.

- Level 3 (L3) — Full order book, also called MBO (Market-by-Order): Every individual order is visible on its own, tagged with its own order ID, at every price level. Every add, modify, cancel, and trade is a separate event tied to one specific order. This is the only level where you can see queue position, literally which order sits ahead of which at a given price.

Each level is a superset of the one below it: L3 can always be collapsed down into L2, and L2 down into L1. But it never works the other way — once data has been aggregated down to L1 or L2, the individual-order detail is gone permanently and can't be recovered.

Anything that depends on seeing individual order behavior needs L3/MBO — things like spoofing or layering detection, queue-position and fill-probability modeling, iceberg-order detection etc. And since the Level 1 data only updates around trades, so it silently misses the vast majority of what actually happens inside the order book.

Ninjatrader tick replay does not fulfil this type of requirement, although that is very useful for doing other type of market analysis.

I hope this clarifies the difference between MBO and Level 1 and Level 2 Data.

Perhaps it might be helpful if someone can tag those members here, who have been working with MBO data earlier.