ChronosServer: real-time access to “native” multi-terabyte retrospective data warehouse by thousands of concurrent clients

Abstract

ChronosServer runs on a cluster of commodity hardware and possess scalability, high availability, and fault tolerance properties. It turns vast amounts of already existing data into actionable intelligence with no changes to the source data files. ChronosServer discovers files on cluster nodes, analyses their structure, and provides format independent SQL-like query model to access their contents. It is capable to read compressed data directly in various formats, including NetCDF, GeoTIFF, GRIB, HDF and many others. This entirely preserves original file metadata as is, vital for its correct interpretation and processing by other software. New data added to the system in a seamless plug-and-play fashion by simple copying it to a cluster node reducing administration overheads. This allows existing software like GIS or statistical packages to operate on files in use by ChronosServer as well as unmodified legacy code to generate data for it. ChronosServer preserves operational infrastructure intact avoiding painful, time- consuming and error-prone data conversions while offering additional opportunities for data analysis.