Wednesday 1st September 2004
Graeme Burnett, Deutsche Bank

HDF5 in Finance

HDF5 is an infrastructure for the supply of large quantities of multidimensional, arbitrary precision data to applications at scalable rates, in a platform independent manner.

The application of statistical analysis techniques in the field of High Frequency Finance requires significant analytical performance and data bandwidth. The former is being delivered by faster processors like the Opteron, but the delivery of data has not kept up with Moore's law and requires an innovative approach.

The traditional approach to data engineering has been delimited files and databases. These technologies suffice for small discrete datasets. Databases are good for data selection using queries on predefined statistical on indexes. They are less than effective at providing a DSS (decision support systems) function and handling time series data.

Flat files can only be delivered to an application unintelligently - it cannot be parallelised easily or transformed automatically by the consumer. Although clustered database technology offers some promise in scalable data delivery, this approach is constrained  by complexity and cost.

HDF5 solves the data engineering needs of the scientific community and will do the same for the financial community by providing a lightweight infrastructure which can overlay existing platforms.

Graeme Burnett is a Software Engineer, Architect and Technology Strategist specialising in applying operational research tools and techniques to the finance industry. He has a strong history delivering innovative, secure, reliable and scalable solutions in financial, government and military environments.

He is currently the architect for the award winning Raptor team, designing and engineering grid-based, ultra-high performance trade routing and analytic software delivered on the Linux platform.

Graeme is a member of the ACM, IEEE and the Operational Research Society.