Using Nextflow to Orchestrate Tightly-coupled HPC Workloads on Rescale

Many of the principles of nextflow such as separation of execution environment and code, the dataflow paradigm, and its ability to work with multiple tools are relatively new to fields that are used to submitting a few tightly coupled HPC jobs in an on-premise cluster using a job scheduler. However, with an increased emphasis on using cloud computing, specialized GPU architectures and experimentation with novel AI/ML tooling, researchers benefit from moving beyond a nest of scripts to a powerful workflow orchestration tool. In this talk we demonstrate the problems we are able to solve using Nextflow to provide "computational pipeline" capabilities in a multi-cloud HPC environment. Example application areas include sequences of tightly coupled HPC jobs in areas such as automotive engineering, molecular dynamics, and materials science.