Missing 'stdout' file when using Slurm and Singularity - slurm

Question

I want to use Nextflow for creating pipelines of singularity components which will be executed by Slurm job scheduler.
I created a simple job to run just a single task:
#!/usr/bin/env nextflow
process createGroups {
executor = 'slurm'
queue = 'upgrade'
memory = '10 GB'
time = '30 min'
cpus = 8
clusterOptions = '--workdir /path/to/the/singularity/image'
"""
singularity run ...
"""
}
The job executes successfully and gives exit status 0, but Nextflow arises "Missing 'stdout' file" error. There is the full output:
Error executing process > 'createGroups'
Caused by:
Missing 'stdout' file: .../work/89/449f849ffc77a967be91ed994da860/.command.out for process > createGroups
Command executed:
executor > slurm (1)
[89/449f84] process > createGroups [100%] 1 of 1, failed: 1 ✘
Error executing process > 'createGroups'
Caused by:
Missing 'stdout' file: .../work/89/449f849ffc77a967be91ed994da860/.command.out for process > createGroups
Command executed:
singularity run ...
Command exit status:
0
Command output:
(empty)
Work dir:
.../work/89/449f849ffc77a967be91ed994da860
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
I couldn't find any solution to fix this error :(
Update:
Using the slurm's --workdir parameter caused that error. I simply used "cd" command to change the path and it solved my problem.

Answer 1

Best to avoid calling Singularity manually, like in your script block. Assuming Singularity will be in your $PATH on every execution/compute node in your queue, then, all you'd need, for example, is:
In your nextflow.config:
process {
executor = 'slurm'
}
singularity {
enabled = true
cacheDir = '/path/to/containers'
}
Then, your nextflow script would look like:
process bwa_mem {
container = 'quay.io/biocontainers/bwa:0.7.17--h5bf99c6_8'
queue = 'upgrade'
cpus = 8
memory = '10 GB'
time = '30.m'
"""
bwa mem
"""
}
See also: https://www.nextflow.io/docs/latest/singularity.html