Little background, as most people here I work mostly with futures . But it looks like I may have to shift to an equities focused team in the next few months, this team will be newly created to trade USA/European Equities. For our futures research we are using R for prototyping then deploying in C++ using RTS. Our futures portfolios usually contain 60-70 global futures instruments. R can handle that amount of data. Mostly looking at mid to long term strats, nothing HFT.

The problem I am considering is 2 fold, first off R/Python both choke on large data sets and are "slow". R slower than Python but I like R as the style is more like mathematics and the package eco system is better(a debate for another day). If I have to back test against thousands of tickers then I am sure its going to be extremely slow, especially when i do event driven back tests instead of purely vectorized.

Next would be the execution platform. We can continue to use RTS and keep translating code from prototype to production. Or go with a more all in 1 system like Deltix. We already have bloomberg which bought RTS so that links up quite well. But I am looking for other options. A big plus would be the ability to generate icebergs, random order qty, and other size obfuscating techniques.

Any advice or recommendations would be helpful.

An ugly fix, but a quick one, is to simply use more hardware. You can use things like grids and clusters to very easily run your tests in parallel. For example, setup a 10 node cluster and just issue commands from the master for those 10 nodes to start backtests, one ticker per node at a time.

This is pretty easy to accomplish and several examples already exist in R.

This would work so long as you can take stand-alone results from each backtest and then compile them into a portfolio after-the-fact, such as one time per day or one time per week. If you need real-time "trade by trade" decision making based on the state of the entire portfolio, then this likely won't help you.

Mike