#!/usr/bin/env bash
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=20
#SBATCH --mem=100G
USEAGE="metascript.sh <wd> <wd1>"
source ~/anaconda2/etc/profile.d/conda.sh
conda activate assembly
wd=$1
wd1=$2
cd $wd
cd $wd1
for f in SRR*/ ; do
[[ -e $f ]] || continue
SRR=${f::-1}
cd ../..
jdid=$(sbatch -J FirstQC_$SRR ./pipelines/preprocessingbowtietrinity/FirstFastqc.sh $wd $wd1 $SRR)
#echo ${jdid[0]}|grep -o '[0-9]\+'
jobid=${jdid[0]}
jobid1=${jobid[0]}|grep -o '[0-9]\+'
#echo $jobid1
Hi all just having issues with my bash scripting, so I can print the line ${jdid[0]}|grep -o '[0-9]+' however when I assign it to a variable it is unable to return anything.
If the idea is to extract just the job ID from the output of sbatch, you can also use sbatch's --parsable argument. See here in the documentation.
jdid=$(sbatch --parsable -J FirstQC_$SRR ./pipelines/preprocessingbowtietrinity/FirstFastqc.sh $wd $wd1 $SRR)
and jdij will only contain the job ID if the cluster is not part of a federation.
jobid1=${jobid[0]}|grep -o '[0-9]\+'
I can print the line ${jdid[0]}|grep -o '[0-9]+' however when I assign it to a variable it is unable to return anything.
In order to assign the output of a pipeline to a variable or insert it into a command line for any other purpose, you have Command Substitution at hand:
jobid1=`echo ${jobid[0]}|grep -o '[0-9]\+'`
Of course, with bash this is better written as:
jobid1=`<<<${jobid[0]} grep -o '[0-9]\+'`
If the issue is printing the line ${jdid[0]}|grep -o '[0-9]+' as your question.
Just put the line in double quotation marks and it will work out.
Here is a little test i made:
jobid1="{jobid[0]}|grep -o '[0-9]\+'"
echo $jobid1
the out put is {jobid[0]}|grep -o '[0-9]\+'
Related
I create this batch file myfirst_slurm_job.sh that contain the following code:
#!/bin/bash
#SBATCH --output="slurm1.txt"
cd $HOME/..
echo $PWD
echo $SLURMD_NODENAME
echo $SLURM_NTASKS
and then I run this command line:
sbatch myfirst_slurm_job.sh
note: it's my first post
You need to specify the --ntasks/-n flag;
#SBATCH -n 1
else SLURM won't bother to define this variable for you.
Hello I'm running this bash script here and the command "bwa index .." will generate several files, and this command doesn't have the "-o" flag for specifying output path. I want to redirect these files into a new folder. I've searched online and all the answers I came across are redirecting files generated to a specific file name, but how to generally just redirect the output files into a new folder? Thanks very much!
#!/bin/bash
### qsub file.name to run from anywhere
### first step of BMW, making indexed fasta file
#PBS -N bwa_index
#PBS -S /bin/bash
#PBS -l walltime=24:00:00
#PBS -l nodes=1:ppn=8
#PBS -l mem=64gb
#PBS -o /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/bwa_index.out
#PBS -e /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/bwa_index.err
date
module load gcc/6.2.0
module load bwa/0.7.17
bwa index -p mm10_bwa_idx -a bwtsw /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/mm10.fa
date
echo END
You can change to the target directory and then run the bwa command. Using pushd and popd will ensure that rest of the script runs in correct directory.
pushd some_directory
bwa index -p mm10_bwa_idx -a bwtsw /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/mm10.fa
popd
First
You can define a variable and use it in your script.
Second
You can pass arguments to your script and capture them via $1, $2, $3 and so on.
Third if you wanted to be fully dynamic by using option ( e.g. --output | -o ) you can use a simple parse as bellow:
#!/bin/bash
################################################################################
# main flags, both longs and shorts
################################################################################
ARGS=`getopt -o "o::" -l "output:" -- "$#"`
eval set -- "$ARGS"
declare -A _output;
_output['flag']=0;
_output['path']=0;
################################################################################
# extract options and their arguments into variables.
################################################################################
while true ; do
case "$1" in
-o | --output )
_output['flag']=1;
_output['path']=$2;
shift 2;
;;
--)
shift;
break;
;;
* )
echo "Internal error!" ; exit 1
;;
esac
done
echo "output: ${_output['path']}";
And usage is super simple, you can use --output <YOUR-PATH> or -o <YOUR-PATH>
# test
./cli.sh --output /path/to/dir
output: /path/to/dir
./cli.sh -o /path/to/dir
output: /path/to/dir
I want to pass a parameter to as bash script in a cluster in order to name the job. I tried this:
#!/bin/bash
#SBATCH -J "$1" #<--- to name the job with the first parameter
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
echo "this is a test job named" $1
Gate main.mac
When I launch the job with
sbatch my_script.sh test_sript
I'm getting a file named $1-23472.out . It appears that "$1" didn't be interpreted. How can I have a file named "test_script-23472.out" ?
Also, is the line Gate main.mac mandatory? Can anyone explains me why we should put it ?
Many thanks
You probably can't do it exactly as you want to, but here's a solution that comes pretty close:
Batch script:
#!/bin/bash
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
echo "this is a test job named" $SLURM_JOB_NAME
(rest of your script here)
Submit with:
$ sbatch -J jobname my_script.sh
Slurm will not interpret the Bash variable in the comments. Bash either since it is in a comment.
One solution is a construct like this for submission:
ARG="<something>" sbatch -J "$ARG" my_script.sh test_sript "$ARG"
As for the Gate main.mac line, it is used to start the Gate program with main.mac as argument.
This is how I've been formatting slurm scripts to parse bash variables as job names.
#!/bin/bash
MYVAR=$1
sbatch --export=ALL -J ${MYVAR} --wrap="run something"
I wanted to indicate the name and other values of some variables of a job from another file but I get an error.
sbatch: error: Unable to open file 10:12:35
file.sh
#!/bin/bash
DATE=`date '+%Y-%m-%d %H:%M:%S'`
name='test__'$DATE
sbatch -J $name -o $name'.out' -e $name'.err' job.sh
job.sh
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --nodes=1 # number of nodes
#SBATCH --ntasks-per-node=2 # number of cores
#SBATCH --output=.out
#SBATCH --error=.err
#module load R
Rscript script.R
script.R
for(i in 1:1e6){print(i)}
You are wrongly quoting the variables and the space requested in the date is creating two arguments to sbatch, hence he is complaining about that wrong parameter.
If I were you, I would avoid the space (as a general rule, cause it is more error prone and always requires quoting):
file.sh:
#!/bin/bash
DATE=$(date '+%Y-%m-%dT%H:%M:%S')
name="test__$DATE"
sbatch -J "$name" -o "${name}.out" -e "${name}.err" job.sh
I have a big script (call it test) that, after stripping out the unrelated parts, comes down to just this using which I can explain my question:
#!/bin/bash
bash -c "$#"
This doesn't work as expected. E.g. ./test echo hi executes the only the echo and the argument disappears!
Testing with various inputs I can see only $1 is passed to bash -c ... and rest are discarded.
But if I use a variable like:
#!/bin/bash
cmd="$#"
bash -c "$cmd"
it works as expected for all inputs.
Questions:
1) I would like to understand why the double quotes don't "pass" the entire command line arguments to bash -c .... What am I missing here (that it works perfectly fine when using an intermediate variable)?
2) Why does bash discard the rest of the arguments (except $1) without any error messages?
For example:
bash -c "ls" -l -a hi hello blah
simply runs echo and hi hello blah doesn't result in any errors at all?
(If possible, please refer to the bash grammar where this behaviour is documented).
1) I would like to understand why the double quotes don't "pass" the entire command line arguments to bash -c .... What am I missing here (that it works perfectly fine when using an intermediate variable)?
From info bash #:
#
($#) Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands
to a separate word. That is, "$#" is equivalent to "$1" "$2" ....
Thus, bash -c "$#" is equivalent to bash -c "$1" "$2" .... In the case of ./test echo hi invocation, the expression is expanded to
bash -c "echo" "hi"
2) Why does bash discard the rest of the arguments (except $1) without any error messages?
Bash actually doesn't discard anything. From man bash:
If the -c option is present, then commands are read from the first non-option argument command_string. If there are arguments after the command_string, they are assigned to the positional parameters, starting with $0.
Thus, for the command bash -c "echo" "hi", Bash passes "hi" as $0 for the "echo" script.
bash -c "ls" -l -a hi hello blah
simply runs echo and hi hello blah doesn't result in any errors at all?
According to the rules mentioned above, Bash executes "ls" script and passes the following positional parameters to this script:
$0: "-l"
$1: "-a"
$2: "hi"
$3: "hello"
$4: "blah"
Thus, the command actually executes ls, and the positional parameters are unused in the script. You can use them by referencing to the positional parameters, e.g.:
$ set -x
$ bash -c "ls \$0 \$1 \$3" -l -a hi hello blah
+ bash -c 'ls $0 $1 $3' -l -a hi hello blah
ls: cannot access hello: No such file or directory
You should be using $* instead of $# to pass command line as string. "$#" expands to multiple quoted arguments and "$*" combines multiple arguments into a single argument.
#!/bin/bash
bash -c "$*"
Problem is with your $# it executes:
bash -c echo hi
But with $* it executes:
bash -c 'echo hi'
When you use:
cmd="$#"
and use: bash -c "$cmd" it does the same thing for you.
Read: What is the difference between “$#” and “$*” in Bash?