Demultiplexing at scale with Arteria

With the advent of new sequencing instruments, high throughput labs face unprecedented volumes of data — exceeding several hundred terabases per year. On top of that, core facilities often have to mix different library preparations and read lengths on a single flowcell, which is not always supported by the manufacturer's software. To manage this avalanche of information, core facilities require not only computational power but also a robust and flexible infrastructure that ensures data quality and uninterrupted flow. In other words, and while compute efficiency is crucial, processing pipelines must also address other critical factors, including robustness, reproducibility, and traceability. From an operator’s perspective, an ideal pipeline should automatically process vanilla runs reliably while offering traceability and customization options for identifying and diagnosing cases requiring human intervention. Enter Arteria [1] — a collection of microservices built around Nextflow pipelines at the National Genomics Infrastructure (NGI) in Uppsala, Sweden. Arteria streamlines the processing, quality control, and delivery of sequencing data from the moment it emerges from the sequencer until it reaches researchers. Key features of Arteria include: - Automated Processing: Arteria decreases processing time, enabling bioinformaticians to focus on analyzing problematic runs. - Nextflow and nf-core Integration: Arteria wraps around Nextflow and nf-core to execute essential steps, including demultiplexing, quality control, and report generation. - Flagging and Prioritization: the system identifies sequencing runs that may need reprocessing or resequencing, ensuring efficient resource allocation. Here, we present Arteria’s architecture, its role in managing sequencing data, and how it empowers our genomics core at NGI by automating and facilitating critical processes in our data workflow. [1] Dahlberg et al., GigaScience 2019, https://doi.org/10.1093/gigascience/giz135