The growing interest in program-based trading in China brings with it an increased awareness of the need for high-performance management of financial market data. However, this isn't the only factor causing an explosion in data volumes in the Chinese market. Indeed the use and management of this growing amount of data extends far beyond the realms of the front office; pre-trade risk management, regulatory compliance and reference data requirements must all be considered when examining how firms can best face up to this challenge. A related issue is that the number of sources and product complexity means that data is best handled by centralised high-performance data management systems that can feed throughout the enterprise.
The data volumes firms must now interact with are huge. Worldwide equity markets alone produce well over three billion records per day, with options and FX at least as large. An essential goal is the ability to keep up with these market volumes, and in particular, with bursts of activity during the day.
These bursts can double the volumes required to be processed under a business-as-usual environment. If financial institutions' systems are not up for the job, this can present significant challenges. Firms must therefore think very carefully about any limits to the amount of data that can be processed. A substantial buffer of performance must be held in reserve to ensure sudden increases in volumes do not bring systems to a grinding halt. Loss of performance on this basis is simply no longer acceptable, and it is both physical hardware as well as software that must pass muster on this point.
All this provides an exciting challenge for China's investment industry to develop or acquire the technology to manage this data effectively.
At the moment, many financial institutions in China either do not keep their own market data at all, relying on third party vendors where necessary, or else store their data in a traditional database format. This can be challenging as it does not support the high-performance queries demanded by algo trading, risk management or complex event processing (CEP). Typically companies have written custom applications using flat files - this works fine for storing data but does not provide the performance required for ad hoc queries. What is needed is some way of holding the data efficiently that permits very fast queries against it.
In planning for this, it is helpful to consider real-time data (current market activity) and historical data (previous market activity) separately. Both are equally as important and each has its own special concerns. For example, even trading algorithms or risk management systems that work solely on real-time data might nevertheless be developed and back-tested on historical information.
Sign up for Computerworld eNewsletters.