I was writing a small shell script in bash to measure the speed of several executables using the time command. Because I wanted to time several different programs, to avoid repeating myself I saved the time command and the format string I wanted and tried to use variable expansion (testing the performance of ls as an example):
#!/usr/bin/env bash
timer="/usr/bin/time -f 'elapsed time: %E' --"
$timer ls 1>/dev/null
The time command is supposed to use the -- to separate its options (in this case a format string) from the command it's supposed to time. However, in this script time insists on trying to execute my format string and fails with
/usr/bin/time cannot run time:: No such file or directory
Command exited with non-zero status 127
'elapsed
But if I put the whole command on one line with no variable expansion it correctly recognizes the format string and executes ls:
#!/usr/bin/env bash
/usr/bin/time -f 'elapsed time: %E' -- ls 1>/dev/null
produces "elapsed time: 0:00.00"
I ended up just typing out the whole command on each line to get it to run, but I was curious if anyone could explain why variable expansion doesn't work here.
The data in $timer is split on $IFS (spaces), and used as separate arguments. It is not expanded and then evaluated as bash code.
Your command $timer ls is therefore equivalent to:
"/usr/bin/time" "-f" "'elapsed time:" "%E'" "--" "ls"
The correct way to do this in your case is using a function:
timer() {
/usr/bin/time -f 'elapsed time: %E' -- "$#"
}
timer ls
PS: This problem is automatically pointed out by shellcheck.
You need to use a BASH array to store command line:
timer=(/usr/bin/time -f 'elapsed time: %E' --)
"${timer[#]}" ls
Related
When I run this script in shell:
printf "Current bash PID is `pgrep -f bash`\n"
using this command:
$ bash script.sh
I get back this output:
Current bash PID is 5430
24390
Every time I run it, I get a different number:
Current bash PID is 5430
24415
Where is the second line coming from?
When you use backticks (or the more modern $(...) syntax for command substitution), you create a subshell. That's a fork()ed-off, independent copy of the shell process which has its own PID, so pgrep finds two separate copies of the shell. (Moreover, pgrep can be finding copies of bash running on the system completely unrelated to the script at hand).
If you want to find the PID of the current copy of bash, you can just look it up directly (printf is better practice than echo when contents can contain backslashes or if the behavior of echo -n or the nonstandard bash extension echo -e is needed, but neither of those things is the case here, so echo is fine):
echo "Current bash PID is $$"
Note that even when executed in a subshell, $$ expands to the PID of the parent shell. With bash 4.0 or newer, you can use $BASHPID to look up the current PID even in a subshell.
See the related question Bash - Two processes for one script
I would like to randomly select one random command in a bash script.
I have tried commands like
echo $(( $RANDOM % 12 ))
But they work in the UNIX command line but not in my bash script.
Anyone have a solution for my problem?
-- Updated answer as per discussion in comments --
For Bash Users:
Code given in question works fine. Just don't forget to set permissions to +x (grants executable permission to all). Eg: chmod +x myscript.sh
Caution: Don't set RANDOM to a value. It looses its special properties.
Bash man page says-
RANDOM Each time this parameter is referenced, a random integer
between 0 and 32767 is generated. The sequence of random numbers may
be initialized by assigning a value to RANDOM. If RANDOM is unset, it
loses its special properties, even if it is sub- sequently reset.
For dash Users:
sh in ubuntu is a symbolic link to dash.
dash does not support RANDOM variable, it is just another variable.
If you are bound to use sh, you can use date function to generate
pseudo randoms only if your script is not time-dependent. (if your
script runs at strict intervals of time, it might produce same values)
RANDOM=`date '+%s'`
echo $(( $RANDOM % 12 )) # To generate random numbers less than 12
In General:
If you want to generate true random numbers use this-
dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -f1 -d" "
Source: http://www.linuxquestions.org/questions/programming-9/shell-script-random-variable-4088/
Create the shell script test.sh
chmod +x test.sh
content of test.sh
#!/bin/bash
echo $(( $RANDOM % 12 ))
Run it via
bash test.sh
or
./test.sh
I'm trying to write a shell script that calls another script that then executes a rsync command.
The second script should run in its own terminal, so I use a gnome-terminal -e "..." command. One of the parameters of this script is a string containing the parameters that should be given to rsync. I put those into single quotes.
Up until here, everything worked fine until one of the rsync parameters was a directory path that contained a space. I tried numerous combinations of ',",\",\' but the script either doesn't run at all or only the first part of the path is taken.
Here's a slightly modified version of the code I'm using
gnome-terminal -t 'Rsync scheduled backup' -e "nice -10 /Scripts/BackupScript/Backup.sh 0 0 '/Scripts/BackupScript/Stamp' '/Scripts/BackupScript/test' '--dry-run -g -o -p -t -R -u --inplace --delete -r -l '\''/media/MyAndroid/Internal storage'\''' "
Within Backup.sh this command is run
rsync $5 "$path"
where the destination $path is calculated from text in Stamp.
How can I achieve these three levels of nested quotations?
These are some question I looked at just now (I've tried other sources earlier as well)
https://unix.stackexchange.com/questions/23347/wrapping-a-command-that-includes-single-and-double-quotes-for-another-command
how to make nested double quotes survive the bash interpreter?
Using multiple layers of quotes in bash
Nested quotes bash
I was unsuccessful in applying the solutions to my problem.
Here is an example. caller.sh uses gnome-terminal to execute foo.sh, which in turn prints all the arguments and then calls rsync with the first argument.
caller.sh:
#!/bin/bash
gnome-terminal -t "TEST" -e "./foo.sh 'long path' arg2 arg3"
foo.sh:
#!/bin/bash
echo $# arguments
for i; do # same as: for i in "$#"; do
echo "$i"
done
rsync "$1" "some other path"
Edit: If $1 contains several parameters to rsync, some of which are long paths, the above won't work, since bash either passes "$1" as one parameter, or $1 as multiple parameters, splitting it without regard to contained quotes.
There is (at least) one workaround, you can trick bash as follows:
caller2.sh:
#!/bin/bash
gnome-terminal -t "TEST" -e "./foo.sh '--option1 --option2 \"long path\"' arg2 arg3"
foo2.sh:
#!/bin/bash
rsync_command="rsync $1"
eval "$rsync_command"
This will do the equivalent of typing rsync --option1 --option2 "long path" on the command line.
WARNING: This hack introduces a security vulnerability, $1 can be crafted to execute multiple commands if the user has any influence whatsoever over the string content (e.g. '--option1 --option2 \"long path\"; echo YOU HAVE BEEN OWNED' will run rsync and then execute the echo command).
Did you try escaping the space in the path with "\ " (no quotes)?
gnome-terminal -t 'Rsync scheduled backup' -e "nice -10 /Scripts/BackupScript/Backup.sh 0 0 '/Scripts/BackupScript/Stamp' '/Scripts/BackupScript/test' '--dry-run -g -o -p -t -R -u --inplace --delete -r -l ''/media/MyAndroid/Internal\ storage''' "
I read the answer for this issue from this link
in Stackoverflow.com. But I am so new in writing shell script that I did something wrong. The following are my scripts:
testscript:
#!/bin/csh -f
pid=$(ps -opid= -C csh testscript1)
while [ -d /proc/$pid ] ; do
sleep 1
done && csh testscript2
exit
testscript1:
#!/bin/csh -f
/usr/bin/firefox
exit
testscript2:
#!/bin/csh -f
echo Done
exit
The purpose is for testscript to call testscript1 first; once testscript1 already finish (which means the firefox called in script1 is closed) testscript will call testscript2. However I got this result after running testscript:
$ csh testscript
Illegal variable name.
Please help me with this issue. Thanks ahead.
I believe this line is not CSH:
pid=$(ps -opid= -C csh testscript1)
In general in csh you define variables like this:
set pid=...
I am not sure what the $() syntax is, perhaps back ticks woudl work as a replacement:
set pid=`ps -opid= -C csh testscript1`
Perhaps you didn't notice that the scripts you found were written for bash, not csh, but
you're trying to process them with the csh interpreter.
It looks like you've misunderstood what the original code was trying to do -- it was
intended to monitor an already-existing process, by looking up its process id using the process name.
You seem to be trying to start the first process from inside the ps command. But
in that case, there's no need for you to do anything so complicated -- all you need
is:
#!/bin/csh
csh testscript1
csh testscript2
Unless you go out of your way to run one of the scripts in the background,
the second script will not run until the first script is finished.
Although this has nothing to do with your problem, csh is more oriented toward
interactive use; for script writing, it's considered a poor choice, so you might be
better off learning bash instead.
Try,
below script will check testscript1's pid, if it is not found then it will execute testscirpt2
sp=$(ps -ef | grep testscript1 | grep -v grep | awk '{print $2}')
/bin/ls -l /proc/ | grep $sp > /dev/null 2>&1 && sleep 0 || /bin/csh testscript2
There are some commands I'd like to run on a grid using qsub (SGE 8.1.3, CentOS 5.9) that need to use a pipe (|) or a redirect (>). For example, let's say I have to parallelize the command
echo 'hello world' > hello.txt
(Obviously a simplified example: in reality I might need to redirect the output of a program like bowtie directly to samtools). If I did:
qsub echo 'hello world' > hello.txt
the resulting content of hello.txt would look like
Your job 123454321 ("echo") has been submitted
Similarly if I used a pipe (echo "hello world" | myprogram), that message is all that would be passed to myprogram, not the actual stdout.
I'm aware I could write a small bash script that each contain the command with the pipe/redirect, and then do qsub ./myscript.sh. However, I'm trying to run many parallelized jobs at the same time using a script, so I'd have to write many such bash scripts each with a slightly different command. When scripting this solution can start to feel very hackish. An example of such a script in Python:
for i, (infile1, infile2, outfile) in enumerate(files):
command = ("bowtie -S %s %s | " +
"samtools view -bS - > %s\n") % (infile1, infile2, outfile)
script = "job" + str(counter) + ".sh"
open(script, "w").write(command)
os.system("chmod 755 %s" % script)
os.system("qsub -cwd ./%s" % script)
This is frustrating for a few reasons, among them that my program can't even delete the many jobXX.sh scripts afterwards to clean up after itself, since I don't know how long the job will be waiting in the queue, and the script has to be there when the job starts.
Is there a way to provide my full echo 'hello world' > hello.txt command to qsub without having to create another file containing the command?
You can do this by turning it into a bash -c command, which lets you put the | in a quoted statement:
qsub bash -c "cmd <options> | cmd2 <options>"
As #spuder has noted in the comments, it seems that in other versions of qsub (not SGE 8.1.3, which I'm using), one can solve the problem with:
echo "cmd <options> | cmd2 <options>" | qsub
as well.
Although my answer is a bit late I am adding it for any incoming viewers. To use a pipe/direct and submit that as a qsub job you need to do a couple of things. But first, using qsub at the end of a pipe like you're doing will only result in one job being sent to the queue (i.e. Your code will run serially rather than get parallelized).
Run qsub with enabling binary mode since the default qsub behavior rather expects compiled code. For that you use the "-b y" flag to qsub and you'll avoid any errors of the sort "command required for a binary mode" or "script length does not match declared length".
echo each call to qsub and then pipe that to shell.
Suppose you have a file params-query.txt which hold several bowtie commands and piped calls to samtools of the following form:
bowtie -q query -1 param1 -2 param2 ... | samtools ...
To send each query as a separate job first prepare your command line units from STDIN through xargs STDIN. Notice the quotes around the braces are important if you are submitting a command of piped parts. That way your entire query is treated a single unit.
cat params-query.txt | xargs -i echo qsub -b y -o output_log -e error_log -N job_name \"{}\" | sh
If that didn't work as expected then you're probably better off generating an intermediate output between bowtie and samtools before calling samtools to accept that intermediate output. You won't need to change the qsub call through xargs but the code in params-query.txt should look like:
bowtie -q query -o intermediate_query_out -1 param1 -2 param2 && samtools read_from_intermediate_query_out
This page has interesting qsub tricks you might like
grep http *.job | awk -F: '{print $1}' | sort -u | xargs -I {} qsub {}