bash - errors trying to pipe commands to run to separate function - linux

I'm trying to get this function for making it easy to parallelize my bash scripts working. The idea is simple; instead of running each command sequentially, I pipe the command I want to run to this function and it does while read line; run the jobs in the bg for me and take care of logistics.... it doesn't work though. I added set -x by where stuff's executed and it looks like I'm getting weird quotes around the stuff I want executed... what should I do?
runParallel () {
while read line
do
while [ "`jobs | wc -l`" -eq 8 ]
do
sleep 2
done
{
set -x
${line}
set +x
} &
done
while [ "`jobs | wc -l`" -gt 0 ]
do
sleep 1
jobs >/dev/null 2>/dev/null
echo sleeping
done
}
for H in `ypcat hosts | grep fmez | grep -v mgmt | cut -d\ -f2 | sort -u`
do
echo 'ping -q -c3 $H 2>/dev/null 1>/dev/null && echo $H - UP || echo $H - DOWN'
done | runParallel
When I run it, I get output like the following:
> ./myscript.sh
+ ping -q -c3 '$H' '2>/dev/null' '1>/dev/null' '&&' echo '$H' - UP '||' echo '$H' - DOWN
Usage: ping [-LRUbdfnqrvVaA] [-c count] [-i interval] [-w deadline]
[-p pattern] [-s packetsize] [-t ttl] [-I interface or address]
[-M mtu discovery hint] [-S sndbuf]
[ -T timestamp option ] [ -Q tos ] [hop1 ...] destination
+ set +x
sleeping
>

The quotes in the set -x output are not the problem, at most they are another result of the problem. The main problem is that ${line} is not the same as eval ${line}.
When a variable is expanded, the resulting words are not treated as shell reserved constructs. And this is expected, it means that eg.
A="some text containing > ; && and other weird stuff"
echo $A
does not shout about invalid syntax but prints the variable value.
But in your function it means that all the words in ${line}, including 2>/dev/null and the like, are passed as arguments to ping, which set -x output nicely shows, and so ping complains.
If you want to execute from variables complicated commandlines with redirections and conditionals, you will have to use eval.

If I'm understanding this correctly, you probably don't want single quotes in your echo command. Single quotes are literal strings, and don't interpret your bash variable $H.

Like many users of GNU Parallel you seem to have written your own parallelizer.
If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:
cat hosts | parallel -j8 'ping -q -c3 {} 2>/dev/null 1>/dev/null && echo {} - UP || echo {} - DOWN'
You can install GNU Parallel simply by:
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
Watch the intro videos for GNU Parallel to learn more:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Put your command in an array.

Related

Different results when running commands in braces within a bash script

I was editing a script and as the script was getting a bit long I decided to enclose the main part of the script in braces and divert the output to a log file instead of having individual log redirects for commands. Then I noticed that a command block that checks for a running copy of the script gives 2 different results depending if it is enclosed in braces.
I run the script as:
$ /bin/bash scriptname.bash
My question is why the same command block returns 2 different results and if it is possible to have the command block work inside the braces.
Below is the command block:
#!/bin/bash
#set -x # Uncomment to debug this shell script
#
##########################################################
# DEFINE FILES AND VARIABLES HERE
##########################################################
THIS_SCRIPT=$(basename $0)
TIMESTAMP=$(date +%Y-%m-%d_%H%M%S)
LOGFILE=process_check_$TIMESTAMP.log
##########################################################
# BEGINNING OF MAIN
##########################################################
{
printf "%s\n" "Checking for currently runnning versions of this script"
MYPID=$$ # Capture this scripts PID
MYOTHERPROCESSES=$(ps -ef | \grep $THIS_SCRIPT | \grep -v $MYPID | \grep -v grep | awk '{print $2}')
if [[ "$MYOTHERPROCESSES" != "" ]]
then
printf "%s\n" "ERROR: Another version of this script is running...exiting!"
exit 2
else
printf "%s\n" "No other versions running...proceeding"
fi
printf "%s\n" "Doing some script stuff..."
exit 0
} | tee -a $LOGFILE 2>&1
# End of script
This is not due to the braces, this is due to the pipe.
When you combine commands with a pipe like command | tee, each side of the pipe is executed in a separate sub-process. Shell commands are therefore executed in a sub-shell. That's this sub-shell that you detect.
PS: avoid constructs like ps | grep -v grep, use pidof or pgrep instead

Bash Script output is always 'ps' when piping to grep from ps regardless of PID results

given an array of pids and the code:
for i in ${listedPids[#]}
do
runningCheck="ps -u $USER | grep $i"
grepRes=(${runningCheck})
if [[ -n $grepRes ]]
then
echo $grepRes
echo $runningCheck
... code not related to the issue
fi
done
Regardless if those pids are active or not; I keep getting 'ps' from echo $grepRes while the output of echo $runningCheck shows up with the correct user name and pid. What am I missing?
Replace
"ps -u $USER | grep $i"
by
$(ps -u $USER | grep $i)
Command Substitution: Bash performs the expansion by executing your command and replacing the command substitution with the standard output of the
command, with any trailing newlines deleted.
I simplified your script and here's what it should look like.
for i in "${listedPids[#]}"
do
grepRes=$(ps --no-heading -p $i)
if [[ -n "$grepRes" ]]
then
echo "$grepRes"
... code not related to the issue
fi
done
An even shorter code could be written using while loop.
ps --noheading -p "${listedPids[#]}" | while read grepRes
do
echo "$grepRes"
... code not related to the issue
done
As alvits and l0b0 pointed out, I made a few syntax errors: grepRes=(${runningCheck}) when I just wanted to execute that line and not turn it to a list, and the fact pipes and redirects don't work in variables. In the end pgrep did the job as I just needed to continue looping till all the background processes ended.
Maybe you could try eval.
runningCheck1="ps -u $USER"
runningCheck2=" | grep $i"
echo $runningCheck1$runningCheck
eval $runningCheck1$runningCheck2

How can I get the command that was executed at the command line?

If I call a script this way:
myScript.sh -a something -b anotherSomething
Within my script is there a way to get the command that called the script?
In my script on the first line I'm trying to use:
lastCommand=!!
echo $lastCommand
But the result is always null.
If I do echo !! the only thing that prints to the console is !!, but from the command line if I do echo !! I get the last command printed.
I've also tried:
echo $BASH_COMMAND
but I'm getting null here as well. Is it because the script is called in a subshell and thus there is no previous command stored in memory for the subshell?
The full command which called the script would be "$0" "$#", that is, the command itself followed by all the arguments quoted. This may not be the exact command which was run, but if the script is idempotent it can be run to get the same result:
$ cat myScript.sh
#!/usr/bin/env bash
printf '%q ' "$0" "$#"
printf '\n'
$ ./myScript.sh -a "foo bar" -b bar
./myScript.sh -a foo\ bar -b bar
Here's my script myScript.sh
#!/bin/bash
temp=`mktemp`
ps --pid $BASHPID -f > $temp
lastCommand=`tail -n 1 $temp | xargs | cut -d ' ' -f 8-`
rm $temp
echo $lastCommand
or
#!/bin/sh
last=`cat /proc/$$/cmdline | xargs -0`
echo $last

BASH script : Integrated document creation hangs

I find that a piece of my bash script causes the hang up. I extract it here :
#!/bin/bash
cat << EndOfFspreadFile >> ./myscript.sh
echo Enter Source Path :
read SRCPATH
FILECNT=`find $SRCPATH/* 2>/dev/null | wc -l`
FILECNTERR=`find $SRCPATH/* 2>&1 | grep "find:" | wc -l`
echo count : $FILECNT
echo problems : $FILECNTERR
EndOfFspreadFile
echo done
This script is expected to just append the script part in the integrated block into myscript.sh file. But it just HANGS !
Thanks !
- Mohamed -
Your $ variables and back quotes will get expanded. You need to escape them in script.
Right now you end up searching the entire filesystem.
Basically, find $SRCPATH/* 2>/dev/null | wc -l gets executed as find /* 2>/dev/null | wc -l
Here is how you can rewrite it (just one line example):
FILECNT=\$(find \$SRCPATH/* 2>/dev/null | wc -l)
By the way, it's easy to find out if you run bash -x <your script>.

How to get the command line args passed to a running process on unix/linux systems?

On SunOS there is pargs command that prints the command line arguments passed to the running process.
Is there is any similar command on other Unix environments?
There are several options:
ps -fp <pid>
cat /proc/<pid>/cmdline | sed -e "s/\x00/ /g"; echo
There is more info in /proc/<pid> on Linux, just have a look.
On other Unixes things might be different. The ps command will work everywhere, the /proc stuff is OS specific. For example on AIX there is no cmdline in /proc.
This will do the trick:
xargs -0 < /proc/<pid>/cmdline
Without the xargs, there will be no spaces between the arguments, because they have been converted to NULs.
Full commandline
For Linux & Unix System you can use ps -ef | grep process_name to get the full command line.
On SunOS systems, if you want to get full command line, you can use
/usr/ucb/ps -auxww | grep -i process_name
To get the full command line you need to become super user.
List of arguments
pargs -a PROCESS_ID
will give a detailed list of arguments passed to a process. It will output the array of arguments in like this:
argv[o]: first argument
argv[1]: second..
argv[*]: and so on..
I didn't find any similar command for Linux, but I would use the following command to get similar output:
tr '\0' '\n' < /proc/<pid>/environ
You can use pgrep with -f (full command line) and -l (long description):
pgrep -l -f PatternOfProcess
This method has a crucial difference with any of the other responses: it works on CygWin, so you can use it to obtain the full command line of any process running under Windows (execute as elevated if you want data about any elevated/admin process). Any other method for doing this on Windows is more awkward ( for example ).
Furthermore: in my tests, the pgrep way has been the only system that worked to obtain the full path for scripts running inside CygWin's python.
On Linux
cat /proc/<pid>/cmdline
outputs the commandline of the process <pid> (command including args) each record terminated by a NUL character.
A Bash Shell Example:
$ mapfile -d '' args < /proc/$$/cmdline
$ echo "#${#args[#]}:" "${args[#]}"
#1: /bin/bash
$ echo $BASH_VERSION
5.0.17(1)-release
Another variant of printing /proc/PID/cmdline with spaces in Linux is:
cat -v /proc/PID/cmdline | sed 's/\^#/\ /g' && echo
In this way cat prints NULL characters as ^# and then you replace them with a space using sed; echo prints a newline.
Rather than using multiple commands to edit the stream, just use one - tr translates one character to another:
tr '\0' ' ' </proc/<pid>/cmdline
ps -eo pid,args prints the PID and the full command line.
You can simply use:
ps -o args= -f -p ProcessPid
In addition to all the above ways to convert the text, if you simply use 'strings', it will make the output on separate lines by default. With the added benefit that it may also prevent any chars that may scramble your terminal from appearing.
Both output in one command:
strings /proc//cmdline /proc//environ
The real question is... is there a way to see the real command line of a process in Linux that has been altered so that the cmdline contains the altered text instead of the actual command that was run.
On Solaris
ps -eo pid,comm
similar can be used on unix like systems.
On Linux, with bash, to output as quoted args so you can edit the command and rerun it
</proc/"${pid}"/cmdline xargs --no-run-if-empty -0 -n1 \
bash -c 'printf "%q " "${1}"' /dev/null; echo
On Solaris, with bash (tested with 3.2.51(1)-release) and without gnu userland:
IFS=$'\002' tmpargs=( $( pargs "${pid}" \
| /usr/bin/sed -n 's/^argv\[[0-9]\{1,\}\]: //gp' \
| tr '\n' '\002' ) )
for tmparg in "${tmpargs[#]}"; do
printf "%q " "$( echo -e "${tmparg}" )"
done; echo
Linux bash Example (paste in terminal):
{
## setup intial args
argv=( /bin/bash -c '{ /usr/bin/sleep 10; echo; }' /dev/null 'BEGIN {system("sleep 2")}' "this is" \
"some" "args "$'\n'" that" $'\000' $'\002' "need" "quot"$'\t'"ing" )
## run in background
"${argv[#]}" &
## recover into eval string that assigns it to argv_recovered
eval_me=$(
printf "argv_recovered=( "
</proc/"${!}"/cmdline xargs --no-run-if-empty -0 -n1 \
bash -c 'printf "%q " "${1}"' /dev/null
printf " )\n"
)
## do eval
eval "${eval_me}"
## verify match
if [ "$( declare -p argv )" == "$( declare -p argv_recovered | sed 's/argv_recovered/argv/' )" ];
then
echo MATCH
else
echo NO MATCH
fi
}
Output:
MATCH
Solaris Bash Example:
{
## setup intial args
argv=( /bin/bash -c '{ /usr/bin/sleep 10; echo; }' /dev/null 'BEGIN {system("sleep 2")}' "this is" \
"some" "args "$'\n'" that" $'\000' $'\002' "need" "quot"$'\t'"ing" )
## run in background
"${argv[#]}" &
pargs "${!}"
ps -fp "${!}"
declare -p tmpargs
eval_me=$(
printf "argv_recovered=( "
IFS=$'\002' tmpargs=( $( pargs "${!}" \
| /usr/bin/sed -n 's/^argv\[[0-9]\{1,\}\]: //gp' \
| tr '\n' '\002' ) )
for tmparg in "${tmpargs[#]}"; do
printf "%q " "$( echo -e "${tmparg}" )"
done; echo
printf " )\n"
)
## do eval
eval "${eval_me}"
## verify match
if [ "$( declare -p argv )" == "$( declare -p argv_recovered | sed 's/argv_recovered/argv/' )" ];
then
echo MATCH
else
echo NO MATCH
fi
}
Output:
MATCH
If you want to get a long-as-possible (not sure what limits there are), similar to Solaris' pargs, you can use this on Linux & OSX:
ps -ww -o pid,command [-p <pid> ... ]
try ps -n in a linux terminal. This will show:
1.All processes RUNNING, their command line and their PIDs
The program intiate the processes.
Afterwards you will know which process to kill

Resources