AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

The ever-growing availability of computing power and sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (http://www.aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA's workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with any simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.

This archive record contains the data to reproduce the figures on engine performance in the section "Event versus polling-based engine" of the paper entitled "AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance". It also includes instructions to reproduce the actual data from scratch using AiiDA v1.1.1 and AiiDA v0.12.5.

Identifier
Source https://archive.materialscloud.org/record/2020.0027/v1
Metadata Access https://archive.materialscloud.org/xml?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:materialscloud.org:343
Provenance
Creator Huber, Sebastiaan P.; Zoupanos, Spyros; Uhrin, Martin; Talirz, Leopold; Kahle, Leonid; Häuselmann, Rico; Gresch, Dominik; Müller, Tiziano; Yakutovich, Aliaksandr V.; Andersen, Casper W.; Ramirez, Francisco F.; Adorf, Carl S.; Gargiulo, Fernando; Kumbhar, Snehal; Passaro, Elsa; Johnston, Conrad; Merkys, Andrius; Cepellotti, Andrea; Mounet, Nicolas; Marzari, Nicola; Kozinsky, Boris; Pizzi, Giovanni
Publisher Materials Cloud
Publication Year 2020
Rights info:eu-repo/semantics/openAccess; Creative Commons Attribution 4.0 International https://creativecommons.org/licenses/by/4.0/legalcode
OpenAccess true
Contact archive(at)materialscloud.org
Representation
Language English
Resource Type Dataset
Discipline Materials Science and Engineering