Seamlessly containerised physics analysis workloads
Category
Contribute
Institutions
PSI
Data type
Particle physics
Field
Physics
Researchers
Clemens Lange
Description
The project aims to provide tools for researchers to make their analysis work reproducible. In particle physics, research is typically performed in larger collaborations where responsibilities for the various tasks are distributed. For example, the preparation of data sets (such as simulation and reconstruction) is done by a group of people while the analysis of such data sets is performed by others. While the former task is reproducible since centralized code repositories and processes are used, this is mostly not the case for the latter. The reasons for that are manifold, one of the main reasons is that analysis work is an iterative process with several steps, software requirements, and intermediate data products. Keeping track of those important details is left to the analysts, who often lack the required technical knowledge. With this project, researchers are provided with the necessary training and tooling to make their analyses reproducible with minimal effort. In addition to holding a two-day training event at PSI, training material for this purpose has been updated and
extended. Furthermore, templates and infrastructure have been developed to automatically build container images, integrating seamlessly with the analysis workflows. These images are then distributed on the computing clusters through newly deployed infrastructure at PSI. The technology used for this purpose called CernVM, specifically the CernVM file system (CVMFS), has originally been developed at CERN. Through this project, we were able to make significant contributions to this project and improve its ease of use and performance.