Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Would you guys be able to generate NTM files from historical tick data (say, in the Ninja .txt format) ? I do not need L1 info, this is just to do tick-by-tick replay for real-time testing of trading strategies (CalculateBarOnClose=false)
Thanks in advance
D.
Can you help answer these questions from other members on NexusFi?
Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.
First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28
For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE
Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)
Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.
To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.
For some purpose, I needed to be able to read .ntd daily files.
The above great work of mrjoe, gomi and dalebru allowed me understanding the structure of these files
In the enclosed document, I have illustrated their findings with AAPL quotes for 2014.
Thanks again to them,
The below R code implements the above specification and allows reading .ntd daily files.
It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.
Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.
On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.
The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
We obtain the following chart:
Let's focus on 2014 :
Chart:
It really looks the same as NT chart:
In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.
I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##
Reading from .nft files within C:\Users\Anil\Documents\NinjaTrader 7\db\minute\SBINEQ\:
Error in as.Date.numeric(vector("integer", 1)) :
'origin' must be supplied
Please pardon me as I am not quite familiar with R.
I have also attached one of the file in my NT db.
EDIT: IT WORKED. Apparently the xts package was not installed. :-P
Elite Membership required to download: 20160609.Last.zip
The below R code implements the above specification and allows reading .ntd daily files.
It provides a function, called ntdFileToXts, which accepts two arguments:
a) directory containing the daily .ntd files for a given security
b) a flag "VERBOSE" (true = all comments written on the console)
It returns an xts element containing the series of quotes for the given security.
I do not pretend that the code is fully optimized. But I hope that it is clear.
Let's take the example of AAPL daily quotes provided by (free) Kinetic End of Day.
On my computer, related .ntd files (containing data from 1990 to 2014) are stored on:
C:\Users\Nicolas\Documents\NinjaTrader 7\db\day\AAPL\ directory.
The 3 following lines use the above-mentioned function to read those files, and generate a xts file:
In order to be sure that there was no mistake in the reading of the data by the R code, I have compared the xts element to the file produced by NT's export function: 100% identical.
I have tested it also with MSFT, ^SP500, CL ##-## and ES ##-##
Nicolas
library(xts)
library(BMS)
library(quantmod)
ntdFileToXts <- function(directory, VERBOSE) {
# This function reads all .ntd files in a given directory, and returns the corresponding xts series
cat("\nReading from .nft files within ",directory,":\n", sep="")
# 1. Some auxiliary functions:
# ----------------------------
skip <- function(length) {
readBin(file.con, raw(), n=length)
}
read.as.numeric.LE <- function(length) {
raw.vector <- readBin(file.con, raw(), n=length)
readBin(raw.vector, numeric(), n=1, endian="little")
}
read.as.numeric.BE <- function(length) {
raw.vector <- readBin(file.con, raw(), n=length)
readBin(raw.vector, numeric(), n=1, endian="big")
}
read.as.unsigned.integer.LE <- function(length) {
raw.vector <- readBin(file.con, raw(), n=length)
as.numeric(paste("0x", paste(rev(raw.vector), collapse=''), sep=""))
}
read.as.unsigned.integer.BE <- function(length) {
raw.vector <- readBin(file.con, raw(), n=length)
as.numeric(paste("0x", paste(raw.vector, collapse=''), sep=""))
}
read.as.binary.mask <- function() {
raw.vector <- readBin(file.con, raw(), n=1)
hex2bin(toString(raw.vector[1]))
}
# 2. Creation of the vectors which will temporarily store quotes,
# before the creation of the xts
# ---------------------------------------------------------------
all.date <- as.Date(vector("integer", 1))
all.open <- vector("numeric", 1)
all.high <- vector("numeric", 1)
all.low <- vector("numeric", 1)
all.close <- vector("numeric", 1)
all.volume <- vector("integer", 1)
# 3. For each ntd file of the directory...
# ----------------------------------------
files <- list.files(path = directory, pattern = "*.ntd")
files <- files[order(files)]
index <- 0
for (file.name0 in files) {
full.file.name <- paste(directory, file.name0, sep="")
cat("Reading", full.file.name, "...\n")
file.con <- file(full.file.name, "rb")
# 4. Reading of the first record
# ------------------------------
index = index + 1
if (VERBOSE) cat("\nRecord #1 (specific structure):\n")
# tick size
tick.size <--read.as.numeric.LE(8)
if (VERBOSE) cat(" Tick size:", tick.size, "\n")
# skip 4 bytes
skip(4)
# nb of records
nb.of.records <- read.as.unsigned.integer.LE(4)
if (VERBOSE) cat(" Nb of records:", nb.of.records, "\n")
# open
open <- read.as.numeric.LE(8)
if (VERBOSE) cat(" Open:", open, "\n")
all.open[index] <- open
# high
high <- read.as.numeric.LE(8)
if (VERBOSE) cat(" High:", high, "\n")
all.high[index] <- high
#low
low <- read.as.numeric.LE(8)
if (VERBOSE) cat(" Low:", low, "\n")
all.low[index] <- low
# close
close <- read.as.numeric.LE(8)
if (VERBOSE) cat(" Close:", close, "\n")
all.close[index] <- close
# date
date <- as.Date(as.POSIXct(read.as.unsigned.integer.LE(8)/10000000, origin="1-01-01"))
if (VERBOSE) cat(" Date:", format(date, format="%Y-%m-%d"), "\n")
all.date[index] <- date
# volume
volume <- read.as.unsigned.integer.LE(8)
if (VERBOSE) cat(" Volume:", format(volume, big.mark=","), "\n")
all.volume[index] <- volume
# 5. Reading of record #2 and the subsequent ones
# -----------------------------------------------
for (record.number in 2:nb.of.records) { #
index <- index + 1
if (VERBOSE) cat("\nRecord #", record.number, ":\n", sep="")
# mask 1
mask1 <- read.as.binary.mask()
if (VERBOSE) cat(" Mask 1:", mask1, "\n")
# volume mask within mask 1
volume.mask <- mask1[2:4]
if (all(volume.mask == c(0,0,1))) {
volume.length <- 1
volume.multiplier <- 1
} else if (all(volume.mask == c(0,1,1))) {
volume.length <- 1
volume.multiplier <- 100
} else if (all(volume.mask == c(1,0,0))) {
volume.length <- 1
volume.multiplier <- 500
} else if (all(volume.mask == c(1,0,1))) {
volume.length <- 1
volume.multiplier <- 1000
} else if (all(volume.mask == c(1,1,0))) {
volume.length <- 2
volume.multiplier <- 1
} else if (all(volume.mask == c(1,1,1))) {
volume.length <- 4
volume.multiplier <- 1
} else if (all(volume.mask == c(0,1,0))) {
volume.length <- 8
volume.multiplier <- 1
} else {
cat("ERROR: volume mask not recognized: ", volume.mask)
}
# open mask within mask 1
open.mask <- mask1[5:6]
if (all(open.mask == c(0,0))) {
open.length <- 0
} else if (all(open.mask == c(0,1))) {
open.length <- 1
open.floor <- 128
} else if (all(open.mask == c(1,0))) {
open.length <- 2
open.floor <- 16384
} else if (all(open.mask == c(1,1))) {
open.length <- 4
open.floor <- 1073741824
} else {
cat("ERROR: open mask not recognized: ", open.mask)
}
# delta time mask within mask 1
delta.time.mask <- mask1[7:8]
if (all(delta.time.mask == c(0,0))) {
delta.time.length <- 0
} else if (all(delta.time.mask == c(0,1))) {
delta.time.length <- 1
} else if (all(delta.time.mask == c(1,0))) {
delta.time.length <- 2
} else if (all(delta.time.mask == c(1,1))) {
delta.time.length <- 4
} else {
cat("ERROR: delta time mask not recognized: ", delta.time.mask)
}
#mask 2
mask2 <- read.as.binary.mask()
if (VERBOSE) cat(" Mask 2:", mask1, "\n")
# low mask within mask 2
low.mask <- mask2[1:2]
if (all(low.mask == c(0,0))) {
low.length <- 0
} else if (all(low.mask == c(0,1))) {
low.length <- 1
} else if (all(low.mask == c(1,0))) {
low.length <- 2
} else if (all(low.mask == c(1,1))) {
low.length <- 4
} else {
cat("ERROR: low mask not recognized: ", low.mask)
}
# high mask within mask 2
high.mask <- mask2[3:4]
if (all(high.mask == c(0,0))) {
high.length <- 0
} else if (all(high.mask == c(0,1))) {
high.length <- 1
} else if (all(high.mask == c(1,0))) {
high.length <- 2
} else if (all(high.mask == c(1,1))) {
high.length <- 4
} else {
cat("ERROR: high mask not recognized: ", high.mask)
}
# close mask within mask 2
close.mask <- mask2[7:8]
if (all(close.mask == c(0,0))) {
close.length <- 0
} else if (all(close.mask == c(0,1))) {
close.length <- 1
} else if (all(close.mask == c(1,0))) {
close.length <- 2
} else if (all(close.mask == c(1,1))) {
close.length <- 4
} else {
cat("ERROR: close mask not recognized: ", close.mask)
}
# delta time and date
delta.time <- ifelse(delta.time.length == 0, 1, read.as.unsigned.integer.BE(delta.time.length))
if (VERBOSE) cat(" Delta time:", delta.time)
date <- date + delta.time;
all.date[index] <- date
if (VERBOSE) cat(" => Date:", format(date, format="%Y-%m-%d"), "\n")
# open delta and open
open.delta <- ifelse(open.length == 0, 0, read.as.unsigned.integer.BE(open.length)-open.floor)
if (VERBOSE) cat(" Open delta:", open.delta)
open <- open + open.delta * tick.size
if (VERBOSE) cat(" => Open:", open, "\n")
all.open[index] <- open
# high delta and high
high.delta <- ifelse(high.length == 0, 0, read.as.unsigned.integer.BE(high.length))
if (VERBOSE) cat(" High delta:", high.delta)
high <- open + high.delta * tick.size
if (VERBOSE) cat(" => High:", high, "\n")
all.high[index] <- high
# low delta and low
low.delta <- ifelse(low.length == 0, 0, read.as.unsigned.integer.BE(low.length))
if (VERBOSE) cat(" Low delta:", low.delta)
low <- open - low.delta * tick.size
if (VERBOSE) cat(" => Low:", low, "\n")
all.low[index] <- low
# close delta and close
close.delta <- ifelse(close.length == 0, 0, read.as.unsigned.integer.BE(close.length))
if (VERBOSE) cat(" Close delta:", close.delta)
close <- low + close.delta * tick.size
if (VERBOSE) cat(" => Close:", close, "\n")
all.close[index] <- close
# volume
volume <- volume.multiplier * read.as.unsigned.integer.BE(volume.length)
if (VERBOSE) cat(" Volume:", format(volume, big.mark=","), "\n")
all.volume[index] <- volume
}
close(file.con)
}
# 6. Creation of xts
# ------------------
quotes <- cbind(all.open, all.high, all.low, all.close, all.volume)
colnames(quotes) <- c("open", "high", "low", "close", "volume")
xts(x=quotes, order.by=all.date)
}
Thanks to MrJoe for the Tick file specification. Here is the additional information needed to read day or minute files, along with some code for reading them.
First, note that under "Address" MrJoe has:
0x10 : Price of the first record in the NTD file [8b IEEE754 LE].
0x18,0x20,0x28 : same data as 0x10 (don't know why)
For day and minute bars
the Open is at 0x10,
the High is at 0x18
the Low is at 0x20
the Close at 0x28
For subsequent records after the header:
The Open is calculated the same as for tick files, from the first 1-byte mask
But for Day and Minute files, there is a second mask byte (mask2) for reading DeltaPrices for High, Low, Close
Bits 01234567
0,1 bitmask for bytes to read for Low relative to Open (must be 0 or negative)
2,3 bitmask for bytes to read for High relative to Open (must be 0 or positive)
4,5 not used
6,7 bitmask for bytes to read for Close relative to Low (must be 0 or positive)
For each pair of bits:
00 no data
01 1B BE (Big Endian)
10 2B BE
11 4B BE
Rules, starting with Open
High is equal to Open or higher (data bits 2,3)
Low is equal to Open or lower (data bits 0,1)
Close is equal to Low or higher (data bits 6,7)
Delta Prices are in the following order: High, Low, Close
1. The first DeltaPrice is for the High, a positive number of ticks ABOVE the Open from the header record
2. The second DeltaPrice is for the Low, a positive number representing the number of ticks BELOW the Open
3. The third DeltaPrice is for the Close, a positive number representing the number of ticks ABOVE the CLOSE.
To use the attached file BruReadNtd.cs, make a test solution and make a test calls. Program.cs is also attached as an example for calling it.