Historical data is one of the most crucial information in trading. It is used for charts construction, backtesting, technical analysis, indicators, algo trading and more. Without historical data traders would only see current bar or would have to leave their software continuously running assuming it can save past data. Historical data is one of the most crucial information in trading. It is used for charts construction, backtesting, technical analysis, indicators, algo trading and more. Without historical data traders would only see current bar or would have to leave their software continuously running assuming it can save past data.
Historical Data Collection
We store historical data for you! Storing historical data can be quite challenging considering not a tick can be lost for the data to be complete. Any hole in your data depreciates data value and lowers it’s practical usability.
We can store everything that comes from data feed provider connection – Tick, Level II, Time & Sales. Based on your chosen subscription we then build higher timeframes or provide complete set of data for specific financial instrument.
Data Storage and Processing
We initially collect tick data as a base to calculate higher timeframes. To be able to calculate higher timeframes such as minute, hour or daily, we need to calculate them from recorded ticks. If we store just the tick data, roughly around 1GB of data is stored per instrument per day. Each day we need to process this data to calculate higher timeframes.
If you select to store 100 instrument, it’s around 100 GB’s of data per day, so after couple of months we are easily running into a scientific field of Big Data. This is only valid for tick data. If we configure the system to store level II data, full order book and time & sales data – amount of collected data is monstrous.
To make all this possible we use the latest cloud and data center technology. Uninterrupted service is an absolute necessity here. Even if we have 99.99% uptime, there is still 0.01% that can cause some real troubles and make holes in data.
To prevent it we are running multiple, multilocation instances of realtime data connections to provider to collect the data. Data is then centrally synchronized and stored.
You can configure services via online interface where you can manage your market data provider connection, configure individual feeds, instrument types and select the instruments you want to store the data for.
Once the instruments are set up, it’s possible to further configure what data should be stored. User can select base timeframe, type of data that should be stored and of course it’s possible to set the period for which is the data stored.
Besides configuration user can work with the stored data. There is a list of instrument types and individual instruments under the each instrument type. User can select type of stored data and visualize the stored data on chart and also in grid.
In case there is a bad tick or wrong price coming from the provider, user can easily edit or delete any stored record. Data can also be imported from and exported to various formats.
As a part of web administration interface user can also view data statistics. There are two types of statistics.
The first informs about service itself - how much data is stored, how much processing power was used to calculate the data, bandwidth used to upload all data via API, current months cost for the services and more.
The second provides more insightful information about the data and provider. Here we can see some providers statistics – provider connection failures and disconnects, what and how much data is missing, latency statistics etc.. Data statistics on the other hand shows more info about the data such average spread.
Pricing much differs and is purely based on configured services. Even if you choose you want to store minute or only end of the day data, we still need to collect the tick data and process it to create higher timeframes. There are no two same instruments, each instrument has different traded volume, some are less volatile some are traded more. Therefore the total amount of stored and processed data varies from instrument to instrument.
Another factors are bandwidth and number of API calls. Some clients may be bombarding our system and requesting too much data to distribute it to multiple sources. Such a client will then require more of our resources than another who would only rarely download the data and create only couple of requests per day.
Because of this, total price you pay depends on the combination of different factors:
- Amount of stored data
- Number of API calls
- Number of backup servers
- Processing power
- Used bandwidth
Custom Connection Support
Is market data provider not supported at the moment?
We can add any market data provider.
To start collecting and processing data from provider we need to connect it first. After you pay the connection setup fee, we will connect your desired provider to our system and we are ready to collect your data.
We support all common connection types and protocols – FIX/API, REST, Websockets.
We will deduct setup fee from the payment for data collection and processing so basically you will not have to pay for the connection once you decide to collect the data.