TSDF (Time Series DataFrame) – A data storage architecture for scalable processing of heterogeneous and geospatial time series

TSDF (Time Series DataFrame) - A data storage architecture for scalable processing of heterogeneous and geospatial time series

Category

Contribute

Institutions

EPFL

Data type

Geospatial time series

Field

Environmental Engineering

Researchers

Martí Bosch

Description

The main objectives of the TSDF project were to develop a new language-agnostic specification to efficiently store irregular (potentially geospatial) time series data and to implement a Python interface to read, write and process data stored in this format. The project was successful in achieving both objectives. First, the "tstore" specification was developed and tested to store data from meteorological stations of different sources. Given that the temporal resolutions varied among sources and sensors, the "tstore" specification allows to efficiently partition the dataset so that the data for each station can be stored separately. This approach removes the need of "aligning" heterogeneous time series data, which can result in numerous empty values. Secondly, a Python library named "tstore" was released, supporting read and write operations from and to columnar data structures based on the Arrow data model, so that the data can be transformed into pandas, polars and dask data frames at zero-copy cost. Additionally, the library features a new data structure named TSDF which allows encapsulating irregular time series into a single pandas data frame object (with geopandas spatial operations support if the source data has spatial information). The TSDF data can be read lazily so that the user can control when the data is actually loaded into memory and the operations are performed.
Scroll to Top