TSDF (Time Series DataFrame) - A data storage architecture for scalable processing of heterogeneous and geospatial time series
Category
Institutions
Data type
Field
Researchers
Abstract
Geospatial time series data are pivotal in environmental sciences, from meteorological stations to mobile sensors. However, the absence of standardized practices due to data heterogeneity and irregularity hinders their processing and storage. This proposal introduces TSDF, a data specification addressing these issues. It offers a flexible framework for accessing, processing, storing, and sharing geospatial time series data. TSDF utilizes Apache Parquet for a binary format, enabling hierarchical structuring of measurements. A Python package will be developed for easy data loading into pandas data frames, simplifying operations like filtering, reducing, and combining data. This work streamlines geospatial time series data processing, enhancing interoperability, reusability, and collaboration.