How to use plus_slurm

Submitting Jobs to the cluster is quite simple using the plus_slurm package.

Make sure you are on a Login Node

Accessing the cluster requires the code to be run on a Login Node as only those can submit jobs to the cluster.

All the code also needs to be on a shared folder so all the Compute Nodes can access the files as well. In general, this is your home-folder and folders you find under /mnt. The specifics depend on the cluster you are using, though

You also need to be in a python environment that has plus_slurm in its dependencies. If you do not now what this means, please take a look at this tutorial.

Define a simple job

The first thing you need to do is define what your job should do. You therefore write a class that derives from plus_slurm.Job. The only things you have to do is to supply a run method.

import plus_slurm

class MyJob(plus_slurm.Job):
    def run(self):
        print('Hello World!')

The job class can be anywhere in your sourcecode tree as long as it is not in the script doing the actual job submission. So, put it in a module (i.e. a .py file) or a package (i.e. a .py file in a folder that also has a __init__.py file).

Define a job that takes arguments

Your job can also take any kind of arguments:

import plus_slurm

class JobWithArgs(plus_slurm.Job):
    def run(self, normal_arg, key_arg='my_default'):
        print('normal_arg=%s\nkey_arg=%s' % (str(normal_arg), str(key_arg)))

Getting and configuring the JobCluster

In order to submit your job, you need to get an instance of plus_slurm.JobCluster. The constructor of this class has a lot of keyword arguments. You can set none, some or all of them. They all have quite sensible defaults:

import plus_slurm

my_jobs = plus_slurm.JobCluster(required_ram='6G')

Now we have a JobCluster that asks for 6GB of RAM per Job.

Adding jobs to the JobCluster

In order to add the jobs, use the plus_slurm.JobCluster.add_job() method:

my_jobs.add_job(MyJob)
my_jobs.add_job(JobWithArgs, 'this_is_the_normal_arg', key_arg='and this the key arg')

Adding multiple jobs with just one call

A common use case of job submission is that you want to run the same job on a number of different combinations of parameters.

Let’s consider a job like this:

class AverageData(plus_slurm.Job):
    def run(self, subject_id, condition, lp_filter_freq)
        ...

And you have a list of subject_ids and conditions:

subject_ids = [
    '19800908igdb',
    '19990909klkl',
    '17560127anpr']

conditions = [
    'visual',
    'auditory']

We want to run the jobs for all combinations of subject_ids and conditions. This is what plus_slurm.PermuteArgument is for:

from plus_slurm import PermuteArgument

my_jobs.add_job(AverageData, PermuteArgument(subject_ids), PermuteArgument(conditions), 30)

This call adds 6 jobs, one for every combination of subject_ids and conditions.

This works for all kinds of arguments (normal ones and keyword arguments).

Submitting the Job

Now, all you need to do is to call submit:

my_jobs.submit()

For more advanced uses take a look at the Reference.