I have been trying to search around for an example of how to use the following option for job dependencies, -d, --dependency=<dependency_list>.
In the documentation, the syntax is shown to be after:job_id[[+time][:jobid[+time]...]]
But I am unable to find any examples of this, and to be honest I find the presentation of the syntax confusing.
I have tried sbatch --dependency=after:123456[+5] myjob.slurm and sbatch --dependency=after:123456+5 myjob.slurm, but this yields the error
sbatch: error: Batch job submission failed: Job dependency problem.
How can I add a dependency to Job B so that it starts X minutes after Job A starts?
The square brackets [...] indicate an optional parameter value and should not appear in the actual parameter value. Try with
sbatch --dependency=after:123456+5 myjob.slurm
With guidance from damienfrancois to exclude the brackets, I tried the following
sbatch --dependency=after:123456:+5 myjob.slurm
Which seems to work beautifully, listing it in the queue as dependent.
EDIT: This is for version 19.05.07
Related
I'm fairly new to snakemake and inherited a kind of huge worflow that consists in a sequence of 17 rules that run in serial.
Each rule takes outputs from the previous rules and uses them to run a python script. Everything has worked great so far except that now I'm trying to improve the worflow since some of the rules can be run in parallel.
A rough example of what I'm trying to achieve, my understanding is that wildcards should allow me to solve this.
grid = [ 10 , 20 ]
rule all:
input:
expand("path/to/C/{grid}/file_C" ,grid = grid)
rule process_A:
input:
path_A = "path/to/A/file_A"
path_B = "path/to/B/{grid}/file_B" # A rule further in the worflow could need a file from a previous rule saved with this structure
params:
grid = lambda wc: wc.get(grid)
output:
path_C = "path/to/C/{grid}/file_C"
script:
"script_A.py"
And inside the script I retrieve the grid size parameter:
grid = snakemake.params.grid
In the end the whole rule process_A should be rerun with grid = 10 and with grid = 20 and save each result to a folder whose path depends on grid also.
I know there are several things wrong with this, but I can't seem to find were to start from to figure this out. The error I'm getting now is:
name 'params' is not defined
Any help as to where to start from?
It would be useful to post the error stack trace of name 'params' is not defined to know exactly what is causing it. For now...
And inside the script I retrieve the grid size parameter:
grid = snakemake.params.grid
I suspect you are mixing the script directive with the shell directive. Probably you want something like:
rule process_A:
input: ...
output: ...
params: ...
script:
"script_A.py"
inside script_A.py snakemake will replace snakemake.params.grid with the actual param value.
Alternatively, write a standalone python script that parses command line arguments and you execute like any other program using the shell directive. (I tend to prefer this solution as it makes things more explicit and easier to debug but it also means more boiler-plate code to write a standalone script).
I received a JCL Error after submitting a job.
20.46.44 JOB08763 $HASP165 WPR062M ENDED AT OZA1 - JCL ERROR CN(INTERNAL).
and in SDSF I am seeing this
How can I fix this (Cancel the job)? What is the reason for this error?
Thanks in advance.
If you are authorized to do so, you can cancel a job in SDSF by putting a C in the "N P" column and pressing the Enter key. But, that's your TSO session (the JobID starts with TSU) and you probably don't want to cancel it. The message you received indicated the job you submitted had a JCL error and ended, so there's no need to cancel it because it's no longer running.
The job shown in the screen shot is your current TSO session; you don't want to cancel this, do you? (BTW, please post text instead of images whenever possible).
The jobname of the one in the screen shot is WPR062 and the jobid is TSU08747. The TSU prefix in the jobid tells you its a TSO session.
The job (not TSO session) in error which gave you this message:
20.46.44 JOB08763 $HASP165 WPR062M ENDED AT OZA1 - JCL ERROR CN(INTERNAL)
has a jobname WPR026M with jobid JOB08763. The JOB prefix tells you its a batch job.
You need to look at the job's output in SDSD to find out what caused the JCL error.
For completeness:
Started task have a jobid prefix of STC.
If your system is configured to allow more than 99'999 active jobids, the prefixes will become single character, i.e. T for TSO sessions, J for batch jobs, and S for started tasks.
As already stated the SDSF output you are looking at is showing your TSO UserID. That is long running and is not the job that is in error.
According to the error message
20.46.44 JOB08763 $HASP165 WPR062M ENDED AT OZA1 - JCL ERROR CN(INTERNAL)
The actual jobname is WPR062M. To investigate the issue I suggest that you use the command PREFIX WMPR062* and then use the H command. The output you are looking for is in the Held queue.
Investigate that job by putting an S in the CmdLine (note, I don't have SDSF on my system but the command column is located on the left side of the screen.
In that job will be the reason for the JCLerror.
I want to find the target branch when a pull request is submitted on GitHub, in my Jenkins pipeline. To achieve this I am doing the following:
I am invoking a windows batch file from my Jenkinsfile, which in turn invokes a nodejs script. This script internally invokes GitHub APIs to get the target branch which is to be set on some variable in Jenkinsfile(code snippet given below):
Jenkinsfile
env.TARGET_BRANCH = bat "GetTargetBranchFromGit.bat ${env.BRANCH_NAME}"
BatchFile:
node getTargetBranchForPR.js %1
But unfortunately, the variable env.TARGET_BRANCH is not getting set to the target branch even though the nodejs script gets the right value. I am in fact not able to return the value from the batch file. Could someone please help me here?
#npocmaka mention is the right way: How to do I get the output of a shell command executed using into a variable from Jenkinsfile (groovy)?
Accodring to Jenkins' documentation.
returnStdout (optional) If checked, standard output from the task is
returned as the step value as a String, rather than being printed to
the build log. (Standard error, if any, will still be printed to the
log.) You will often want to call .trim() on the result to strip off a
trailing newline.
So your code should look like
env.TARGET_BRANCH = bat( script: "GetTargetBranchFromGit.bat ${env.BRANCH_NAME}",
returnStdout: true
).trim()
If you get back more than expected you probably need to parse it.
I'd like to send myself a text when a job is finished. I understand how to change the job name so that the .o and .e files have the appropriate name. But I'm not sure if there's a way to change the job ID from a string of numbers to a specified key so I know which job it is. I usually have a lot of different jobs going at once, so it's difficult to remember all the different job ID numbers. Is there a way in the .pbs script to change the job ID so that when I get the message I can see which job it is rather than just a string of numbers?
If you are using Torque and add the -N flag, then you can add a name to the job. It will still use the numeric portion of the job id as part of the output and error filenames, but this allows you to add something to help you distinguish among your jobs. For example:
$ echo ls | qsub -N whatevernameyouplease
I have a long Cucumber scenario with only a few steps implemented, and many steps are pending. In my set up, Cucumber skips all pending steps, but still prints all the names of those skipped tests in the output, and that makes it tedious to scroll up to find the step where the scenario failed.
How can I make sure the skipped steps are not printed in the output? Ideally, I want the output to show all passing steps and the last failed step.
I think there is no out of the box formatter that does exactly that. You would need to create a custom one but perhaps you could use the progress formatter. It prints information about the failed steps, green dots for the passing steps and dashes for the skipped ones:
>> cucumber --format progress
..............................F-
(::) failed steps (::)
"Error messages here"
Failing Scenarios:
cucumber features/dashboard.feature:13 # Scenario: Dashboard data is loaded
4 scenarios (1 failed, 3 passed)
26 steps (1 failed, 1 skipped, 24 passed)
0m41.604s
Also you can combine different formatters. One to give its output in the console and others to output its format in different files for later inspection. for example:
$ cucumber -f progress -f pretty --out results.txt -f rerun --out rerun.txt