Running on HPC

Metabarcoding studies often have dozens or hundreds of samples, and some steps of PEMA need a lot of computing power or memory (RAM), e.g. the taxonomy assignment of COI data. An HPC system or a large server is the best place to run PEMA.

With Nextflow, running PEMA on an HPC system is the same command as on a laptop, with two profiles:

• a container engine available on the cluster, usually Singularity or Apptainer (Docker is rarely allowed on HPC systems);
• optionally, the scheduler of the cluster, so that every step of PEMA runs as a separate job.

Singularity or Apptainer

Check that one of them is available, e.g. with

singularity --version     # or: apptainer --version

(on many systems, after module load singularity or similar). If neither is available, ask your system administrators.

Nextflow converts the containers of PEMA to Singularity images the first time they are used. Tell it where to keep them, so they are downloaded only once; for example in your ~/.bashrc:

export NXF_SINGULARITY_CACHEDIR=$HOME/singularity_images     # or NXF_APPTAINER_CACHEDIR

Then run PEMA with the singularity (or apptainer) profile:

nextflow run pema.nf --params-file my_analysis.yaml -profile singularity

Running on one node

The simplest way: submit a single job that runs PEMA on one node, with all the steps running on that node. Set threads in the parameters file to the number of cores you ask for. With SLURM:

#!/bin/bash

#SBATCH --job-name=pema
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=20
#SBATCH --mem=64G
#SBATCH --time=24:00:00
#SBATCH --output=pema.%j.out

module load singularity      # if needed on your system

cd /path/to/pema
nextflow run pema.nf --params-file my_analysis.yaml -profile singularity

Letting Nextflow submit the jobs

Alternatively, Nextflow can submit every step of PEMA as a separate SLURM job, so that the steps of different samples run on different nodes at the same time. Add the slurm profile, and the partition (queue) to use:

nextflow run pema.nf --params-file my_analysis.yaml -profile singularity,slurm -process.queue <partition>

Nextflow itself has to keep running until the end; run it in a screen or tmux session on the login node (if allowed), or in a small, long job (1 core, 4 GB of memory).

Nextflow supports many other schedulers (PBS/Torque, SGE, LSF, HTCondor, …) and cloud platforms; see the Nextflow executors. You can set one in a small configuration file:

// my_cluster.config
process.executor = 'pbspro'
process.queue    = 'workq'

and add -c my_cluster.config to the command.

Memory

RDPClassifier needs more memory for large databases: set memory in the rdp section of the parameters file (for COI with MIDORI, at least 20 GB), and train.memory when training a database (training on MIDORI needed more than 50 GB). When a step is killed for using too much memory, Nextflow retries it once and then stops; raise the memory in the parameters file and run again with -resume, so the steps that were already done are not repeated.