Compute resources

Compute resources

nextPYP can be run in standalone mode or in an HPC environment using the SLURM workload manager. Standalone mode is simpler to setup and can be used to process small to medium sized datasets. For most datasets, however, an instance with access to a SLURM cluster will enable significantly faster processing.

All operations in nextPYP are executed on a per-micrograph or per-tilt-series basis. As a consequence, compute resources are specified per micrograph/tilt-series (independent of the total number of micrographs/tilt-series in a dataset).

Threads, tasks, and arrays

There are three types of processes used in nextPYP:

  • Thread: Single-threaded process running on one CPU core (or hyper-threaded core)

  • Task: Set of threads used to process a single micrograph or tilt-series

  • Array: Set of tasks used to process an entire dataset

Most jobs in nextPYP have a Launch, Split and Merge phases. The Launch phase is typically a lightweigth job used to initiate data processing. In the Split phase, multiple tasks are launched and executed in parallel (one task for each micrograph/tilt-series), while during the Merge phase information from the Split phase is condensed, for example, to produce a single 3D reconstruction from all micrographs or tilt-series in a dataset. Since each phase has different computational requirements, resources are allocated separately for each of them.

Real time information about jobs can be found in the Jobs panel.

Each processing block in nextPYP has a Resources tab that allows specifying resources for each job phase:

Job submission options

The Resources tab consists of three sections, one for each phase (Launch, Split and Merge):


Users are responsible for ensuring that the combination of resources requested is available in the HPC environment where nextPYP is running.


To get information on the status of a job, go to the Jobs panel and click on the icon next to the job.