so I have this code:
uthreads=4
x=1
cd ./wxWidgets/wxWidgets/build-win
for upxs in $(ls ./lib/*.dll)
do
while [ true ] ; do
if [ $x -lt $uthreads ] ; then
x=$((x+1))
(upx -qqq --best --color $upxs; x=$((x-1))) &
break
fi
sleep 1
done
done
x=$((x=1))
the problem lies in the variable being modified in parenthesis. Naturally this does not work as intended as the variable never gets sent back to the parent shell. So my question is how do you do this? The variable must be incremented after the upx command finishes regardless of it's exit status. so a simple && or || won't work here. and you can't use a single & here after upx; because then the process runs in the background and instantly changes the variable...
So I'm effectively stuck and could use some help...
first of all, this: while [ true ] is wrong, true is a command and so is [ you don't need both here just do while true; do (and use [[ in bash, the other is just for POSIX-sh).
this: for upxs in $(ls ./lib/*.dll) is also wrong, and it's more serious; I'll give you some links with information about all that but for now just do: for upxs in ./lib/*.dll
another important mistake is that you're not quoting your variables, EVERY SINGLE VARIABLE MUST BE DOUBLE QUOTED, if there's anything you learn from this post please let it be that.
now regarding your particular problem the whole approach is wrong, you can't do what you're trying to do because you don't know which process will end first and you wait for a particular one or wait for all of them.
also notice you're running four processes with the same file, I'm sure your intention was to process 4 different files in parallel and you don't wait for them to finish either, chances are after your script ends your processes get killed.
the suboptimal but easy solution would be to have 4 processes running in parallel and wait for all of them to finish before starting 4 more, that is accomplished by this (which is similar to your current approach):
#!/bin/bash
uthreads=4
cd ./wxWidgets/wxWidgets/build-win
upxs=(lib/*.dll)
i=0
while ((i<${#upxs[#]})); do
for ((x=0; x<uthreads; x++)); do
[[ ${upxs[i+x]} ]] || break
upx -qqq --best --color "${upxs[i+x]}" &
done
i=$((i+x))
wait
done
now, this might not be enough to satisfy you :) so the way to implement 4 running processes at all times would be this (using PIDs):
#!/bin/bash
uthreads=4
cd ./wxWidgets/wxWidgets/build-win
upxs=(lib/*.dll)
i=0 pids=()
while [[ ${upxs[i]} ]]; do
while ((${#pids[#]}<uthreads)); do
upx -qqq --best --color "${upxs[i]}" & pids+=($!)
[[ ${upxs[++i]} ]] || break
done
sleep .1
pids=($(for pid in "${pids[#]}"; do kill -0 $pid && echo $pid; done))
done
wait
now here's the links I promised:
http://mywiki.wooledge.org/BashPitfalls#if_.5Bgrep_foo_myfile.5D
http://mywiki.wooledge.org/BashFAQ/001
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/ProcessManagement
Assuming that you're trying to 'gate' the activity of a program by controlling how many copies of it are running in the background, check out the xargs command. My system doesn't have upx installed, so the best I can recommend as a code sample is
uthreads=4
cd ./wxWidgets/wxWidgets/build-win
ls ./lib/*.dll | xargs -I{} -P $uthreads upx -qqq --best --color {}
^ ^ ^ ^ ^ ^
+>your list | | | | |
| | | | +> item from your lst
| | | +> your cmd's opts
| | +> your cmd
| + xargs opt for # of processes
+ xargs option to indicate what string to replace with
use xargs --help to see all the options, man xargs if you're lucky for in-depth detail on usage, and search on web and here for more examples of usage. The wikipedia article is a good start.
I hope this helps
Related
I use the following script to check whether wget has finished downloading. To check for this, I'm looking for its PID, and when it is not found the computer shutdowns. This works fine for a single instance of wget, however, I'd like the script to look for all already running wget programs.
#!/bin/bash
while kill -0 $(pidof wget) 2> /dev/null; do
for i in '-' '/' '|' '\'
do
echo -ne "\b$i"
sleep 0.1
done
done
poweroff
EDIT: I'd would be great if the script would check if at least one instance of wget is running and only then check whether wget has finished and shutdown the computer.
In addition to the other answers, you can satisfy your check for at least one wget pid by initially reading the result of pidof wget into an array, for example:
pids=($(pidof wget))
if ((${#pids[#]} > 0)); then
# do your loop
fi
This also brings up a way to routinely monitor the remaining pids as each wget operation completes, for example,
edit
npids=${#pids[#]} ## save original number of pids
while (( ${#pids[#]} -gt 0 )); do ## while pids remain
for ((i = 0; i < npids; i++)); do ## loop, checking remaining pids
kill -0 ${pids[i]} || pids[$i]= ## if not unset in array
done
## do your sleep and spin
done
poweroff
There are probably many more ways to do it. This is just one that came to mind.
I don't think kill is a right Idea,
may be some thing on the lines like this
while [ 1 ]
do
live_wgets=0
for pid in `ps -ef | grep wget| awk '{print $2}'` ; # Adjust the grep
do
live_wgets=$((live_wgets+1))
done
if test $live_wgets -eq 0; then # shutdown
sudo poweroff; # or whatever that suits
fi
sleep 5; # wait for sometime
done
You can adapt your script in the following way:
#!/bin/bash
spin[0]="-"
spin[1]="\\"
spin[2]="|"
spin[3]="/"
DOWNLOAD=`ps -ef | grep wget | grep -v grep`
while [ -n "$DOWNLOAD" ]; do
for i in "${spin[#]}"
do
DOWNLOAD=`ps -ef | grep wget | grep -v grep`
echo -ne "\b$i"
sleep 0.1
done
done
sudo poweroff
However I would recommend using cron instead of an active waiting approach or even use wait
How to wait in bash for several subprocesses to finish and return exit code !=0 when any subprocess ends with code !=0?
I have a command in a variable in Bash:
check_ifrunning=\`ps aux | grep "programmname" | grep -v "grep" | wc -l\`
The command checks if a specific program is running at the moment.
Later in my script, I want to query the value of the variable on a point.
If the specific program is running, the script should sleep for 15 minutes.
I solved it like this:
while [ $check_ifrunning -eq 1 ]; do
sleep 300
done
Will the script execute the command in the variable for each single loop-run or will the value in the variable stay after the first execution?
I have more variables in my script which can change their value. This was just one simple example of this.
Notice that check_ifrunning is set only once, in
check_ifrunning=`ps aux | grep "programmname" | grep -v "grep" | wc -l`
and that it is set before the loop:
while [ $check_ifrunning -eq 1 ]; do
sleep 300
done
You could add, for debugging purposes, an echo check_ifrunning is $check_ifrunning statement inside your while loop just before the sleep ...
You probably simply want (using pidof(8)) - without defining or using any check_ifrunning Bash variable:
while [ -n "$(pidof programname)" ]; do
sleep 300
done
Because you want to test if programname is running at every start of the loop!
You should use the more nestable and more readable $(...) instead of backquotes.
Consider reading the Advanced Bash Scripting Guide...
If you are writing a Bash script, consider to start it with
#!/bin/bash -vx
while debugging. When you are satisfied, remove the -vx...
If you want to encapsulate your commands, the proper way to do that is a function.
running () {
ps aux | grep "$1" | grep -q -v grep
}
With grep -q you get the result as the exit code, not as output; you use it simply like
if running "$programname"; then
:
Ideally, the second grep is unnecessary, but I did not want to complicate the code too much. It still won't work correctly if you are looking for grep. The proper solution is pidof.
See also http://mywiki.wooledge.org/BashFAQ/050
I am using a bash script that calls multiple processes which have to start up in a particular order, and certain actions have to be completed (they then print out certain messages to the logs) before the next one can be started. The bash script has the following code which works really well for most cases:
tail -Fn +1 "$log_file" | while read line; do
if echo "$line" | grep -qEi "$search_text"; then
echo "[INFO] $process_name process started up successfully"
pkill -9 -P $$ tail
return 0
elif echo "$line" | grep -qEi '^error\b'; then
echo "[INFO] ERROR or Exception is thrown listed below. $process_name process startup aborted"
echo " ($line) "
echo "[INFO] Please check $process_name process log file=$log_file for problems"
pkill -9 -P $$ tail
return 1
fi
done
However, when we set the processes to print logging in DEBUG mode, they print so much logging that this script cannot keep up, and it takes about 15 minutes after the process is complete for the bash script to catch up. Is there a way of optimizing this, like changing 'while read line' to 'while read 100 lines', or something like that?
How about not forking up to two grep processes per log line?
tail -Fn +1 "$log_file" | grep -Ei "$search_text|^error\b" | while read line; do
So one long running grep process shall do preprocessing if you will.
Edit: As noted in the comments, it is safer to add --line-buffered to the grep invocation.
Some tips relevant for this script:
Checking that the service is doing its job is a much better check for daemon startup than looking at the log output
You can use grep ... <<<"$line" to execute fewer echos.
You can use tail -f | grep -q ... to avoid the while loop by stopping as soon as there's a matching line.
If you can avoid -i on grep it might be significantly faster to process the input.
Thou shalt not kill -9.
I have a simple bash script, which forms part of an in house web app that I've developed.
It's purpose is to automate deletion of thumbnails of images when the original image has been deleted by the user.
The script logs some basic status info to a file /var/log/images.log
#!/bin/bash
cd $thumbpath
filecount=0
# Purge extraneous thumbs
find . -type f | while read file
do
if [ ! -f "$imagepath/$file" ]
then
filecount=$[$filecount+1]
rm -f "$file"
fi
done
echo `date`: $filecount extraneous thumbs removed>>/var/log/images.log
Whilst the script correctly deletes thumbs, it doesn't correctly output the number of thumbs that are being purged, it always shows 0.
For example, having just manually created some orphaned thumbnails, and then running my script, the manually generated orphaned thumbs are deleted, but the log shows:
Thu Jun 9 23:30:12 BST 2011: 0 extraneous thumbs removed
What am I doing wrong that is stopping $filecounter from showing a number other than zero, when files are being deleted.
I've created the following bash script to test this, and this works perfectly, outputting 0 then 1:
#!/bin/bash
count=0
echo $count
count=$[$count+1]
echo $count
Edit:
Thanks for the answers, but why does the following work
$ x=3
$ x=$[$x+1]
$ echo $x
4
...and also the second example works, yet it doesn't work in the first script?
Second Edit:
This works
count=0
echo Initial Value $count
for i in `seq 1 5`
do
count=$[$count+1]
echo $count
done
echo Final Value $count
Initial Value 0
1
2
3
4
5
Final Value 5
as does replacing count=$[$count+1] with count=$((count+1)), but not in my initial script.
You're using the wrong operator. Try using $(( ... )) instead, e.g.:
$ x=4
$ y=$((x + 1))
$ echo $y
5
$
EDIT
The other problem you're bumping into is down to the pipe. Bumped into this one before (with ksh, but wouldn't suprise me to find that other shells have the same problem). The pipe is forking another bash process, so when you do the increment, filcount is getting incremented in the subshell that's been forked after the pipe. This value isn't passed back to the calling shell as the subshell has it's own independent environment (environment variables are inherited in called processes, but called process cannot modify the environment of the calling process).
As an example, this demonstrates that filecount gets incremented okay:
#!/bin/bash
filecount=0
ls /bin | while read x
do
filecount=$((filecount + 1))
echo $filecount
done
echo $filecount
...so you should see filecount increase in the loop, but the final filecount will be zero because this echo belongs to the main shell, but the forked subshell (which consists purely of the while loop).
One way you can get the value back is like this...
#!/bin/bash
filecount=0
filecount=`ls /bin | while read x
do
filecount=$((filecount + 1))
echo $filecount
done | tail -1`
echo $filecount
This will only work if you don't care about any other stdout output in the loop as this throws it all away apart from the last line we output (the final value of filecount). This works because we're using stdout and stdin to feed the data back to the parent shell.
Depending on your viewpoint this is either a nasty hack or a nifty bit of shell jiggery-pokery. I'll leave you to decide what you think it is :-)
If you remove the pipeline into the while construct, you remove bash's need to create a subshell.
Change this:
filecount=0
find . -type f | while read file; do
if [ ! -f "$imagepath/$file" ]; then
filecount=$[$filecount+1]
rm -f "$file"
fi
done
echo $filecount
to this:
filecount=0
while read file; do
if [ ! -f "$imagepath/$file" ]; then
rm -f "$file" && (( filecount++ ))
fi
done < <(find . -type f)
echo $filecount
That is harder to read because the find command is hidden at the end. Another possibility is:
files=$( find . -type f )
while ...; do
:
done <<< "$files"
Chris J is quite right that you are using the wrong operator and POSIX subshell variable scoping means you can't get a final count that way.
As a side note, when doing math operations you could also consider using the let shell bultin like this:
$ filecount=4
$ let filecount=$filecount+1
$ echo $filecount
5
Also if you want scoping to just work like you expected it to in spite of that pipeline, you could use zsh instead of bash. In this case it should be a drop in replacement and work as expected.
I work with some log system which creates a log file every hour, like follows:
SoftwareLog.2010-08-01-08
SoftwareLog.2010-08-01-09
SoftwareLog.2010-08-01-10
I'm trying to tail to follow the latest log file giving a pattern (e.g. SoftwareLog*) and I realize there's:
tail -F (tail --follow=name --retry)
but that only follow one specific name - and these have different names by date and hour. I tried something like:
tail --follow=name --retry SoftwareLog*(.om[1])
but the wildcard statement is resoved before it gets passed to tail and doesn't re-execute everytime tail retries.
Any suggestions?
I believe the simplest solution is as follows:
tail -f `ls -tr | tail -n 1`
Now, if your directory contains other log files like "SystemLog" and you only want the latest "SoftwareLog" file, then you would simply include a grep as follows:
tail -f `ls -tr | grep SoftwareLog | tail -n 1`
[Edit: after a quick googling for a tool]
You might want to try out multitail - http://www.vanheusden.com/multitail/
If you want to stick with Dennis Williamson's answer (and I've +1'ed him accordingly) here are the blanks filled in for you.
In your shell, run the following script (or it's zsh equivalent, I whipped this up in bash before I saw the zsh tag):
#!/bin/bash
TARGET_DIR="some/logfiles/"
SYMLINK_FILE="SoftwareLog.latest"
SYMLINK_PATH="$TARGET_DIR/$SYMLINK_FILE"
function getLastModifiedFile {
echo $(ls -t "$TARGET_DIR" | grep -v "$SYMLINK_FILE" | head -1)
}
function getCurrentlySymlinkedFile {
if [[ -h $SYMLINK_PATH ]]
then
echo $(ls -l $SYMLINK_PATH | awk '{print $NF}')
else
echo ""
fi
}
symlinkedFile=$(getCurrentlySymlinkedFile)
while true
do
sleep 10
lastModified=$(getLastModifiedFile)
if [[ $symlinkedFile != $lastModified ]]
then
ln -nsf $lastModified $SYMLINK_PATH
symlinkedFile=$lastModified
fi
done
Background that process using the normal method (again, I don't know zsh, so it might be different)...
./updateSymlink.sh 2>&1 > /dev/null
Then tail -F $SYMLINK_PATH so that the tail hands the changing of the symbolic link or a rotation of the file.
This is slightly convoluted, but I don't know of another way to do this with tail. If anyone else knows of a utility that handles this, then let them step forward because I'd love to see it myself too - applications like Jetty by default do logs this way and I always script up a symlinking script run on a cron to compensate for it.
[Edit: Removed an erroneous 'j' from the end of one of the lines. You also had a bad variable name "lastModifiedFile" didn't exist, the proper name that you set is "lastModified"]
I haven't tested this, but an approach that may work would be to run a background process that creates and updates a symlink to the latest log file and then you would tail -f (or tail -F) the symlink.
#!/bin/bash
PATTERN="$1"
# Try to make sure sub-shells exit when we do.
trap "kill -9 -- -$BASHPID" SIGINT SIGTERM EXIT
PID=0
OLD_FILES=""
while true; do
FILES="$(echo $PATTERN)"
if test "$FILES" != "$OLD_FILES"; then
if test "$PID" != "0"; then
kill $PID
PID=0
fi
if test "$FILES" != "$PATTERN" || test -f "$PATTERN"; then
tail --pid=$$ -n 0 -F $PATTERN &
PID=$!
fi
fi
OLD_FILES="$FILES"
sleep 1
done
Then run it as: tail.sh 'SoftwareLog*'
The script will lose some log lines if the logs are written to between checks. But at least it's a single script, with no symlinks required.
We have daily rotating log files as: /var/log/grails/customer-2020-01-03.log. To tail the latest one, the following command worked fine for me:
tail -f /var/log/grails/customer-`date +'%Y-%m-%d'`.log
(NOTE: no space after the + sign in the expression)
So, for you, the following should work (if you are in the same directory of the logs):
tail -f SoftwareLog.`date +'%Y-%m-%d-%H'`
I believe the easiest way is to use tail with ls and head, try something like this
tail -f `ls -t SoftwareLog* | head -1`