I have a simple bash script, which forms part of an in house web app that I've developed.
It's purpose is to automate deletion of thumbnails of images when the original image has been deleted by the user.
The script logs some basic status info to a file /var/log/images.log
#!/bin/bash
cd $thumbpath
filecount=0
# Purge extraneous thumbs
find . -type f | while read file
do
if [ ! -f "$imagepath/$file" ]
then
filecount=$[$filecount+1]
rm -f "$file"
fi
done
echo `date`: $filecount extraneous thumbs removed>>/var/log/images.log
Whilst the script correctly deletes thumbs, it doesn't correctly output the number of thumbs that are being purged, it always shows 0.
For example, having just manually created some orphaned thumbnails, and then running my script, the manually generated orphaned thumbs are deleted, but the log shows:
Thu Jun 9 23:30:12 BST 2011: 0 extraneous thumbs removed
What am I doing wrong that is stopping $filecounter from showing a number other than zero, when files are being deleted.
I've created the following bash script to test this, and this works perfectly, outputting 0 then 1:
#!/bin/bash
count=0
echo $count
count=$[$count+1]
echo $count
Edit:
Thanks for the answers, but why does the following work
$ x=3
$ x=$[$x+1]
$ echo $x
4
...and also the second example works, yet it doesn't work in the first script?
Second Edit:
This works
count=0
echo Initial Value $count
for i in `seq 1 5`
do
count=$[$count+1]
echo $count
done
echo Final Value $count
Initial Value 0
1
2
3
4
5
Final Value 5
as does replacing count=$[$count+1] with count=$((count+1)), but not in my initial script.
You're using the wrong operator. Try using $(( ... )) instead, e.g.:
$ x=4
$ y=$((x + 1))
$ echo $y
5
$
EDIT
The other problem you're bumping into is down to the pipe. Bumped into this one before (with ksh, but wouldn't suprise me to find that other shells have the same problem). The pipe is forking another bash process, so when you do the increment, filcount is getting incremented in the subshell that's been forked after the pipe. This value isn't passed back to the calling shell as the subshell has it's own independent environment (environment variables are inherited in called processes, but called process cannot modify the environment of the calling process).
As an example, this demonstrates that filecount gets incremented okay:
#!/bin/bash
filecount=0
ls /bin | while read x
do
filecount=$((filecount + 1))
echo $filecount
done
echo $filecount
...so you should see filecount increase in the loop, but the final filecount will be zero because this echo belongs to the main shell, but the forked subshell (which consists purely of the while loop).
One way you can get the value back is like this...
#!/bin/bash
filecount=0
filecount=`ls /bin | while read x
do
filecount=$((filecount + 1))
echo $filecount
done | tail -1`
echo $filecount
This will only work if you don't care about any other stdout output in the loop as this throws it all away apart from the last line we output (the final value of filecount). This works because we're using stdout and stdin to feed the data back to the parent shell.
Depending on your viewpoint this is either a nasty hack or a nifty bit of shell jiggery-pokery. I'll leave you to decide what you think it is :-)
If you remove the pipeline into the while construct, you remove bash's need to create a subshell.
Change this:
filecount=0
find . -type f | while read file; do
if [ ! -f "$imagepath/$file" ]; then
filecount=$[$filecount+1]
rm -f "$file"
fi
done
echo $filecount
to this:
filecount=0
while read file; do
if [ ! -f "$imagepath/$file" ]; then
rm -f "$file" && (( filecount++ ))
fi
done < <(find . -type f)
echo $filecount
That is harder to read because the find command is hidden at the end. Another possibility is:
files=$( find . -type f )
while ...; do
:
done <<< "$files"
Chris J is quite right that you are using the wrong operator and POSIX subshell variable scoping means you can't get a final count that way.
As a side note, when doing math operations you could also consider using the let shell bultin like this:
$ filecount=4
$ let filecount=$filecount+1
$ echo $filecount
5
Also if you want scoping to just work like you expected it to in spite of that pipeline, you could use zsh instead of bash. In this case it should be a drop in replacement and work as expected.
Related
So guys,
I need your help trying to identify the fastest and the most "fault" tolerant solution to my problem.
I have a shell script which executes some functions, based on a txt file, in which I have a list of files.
The list can contain from 1 file to X files.
What I would like to do is iterate over the content of the file and execute my scripts for only 4 items out of the file.
Once the functions have been executed for these 4 files, go over to the next 4 .... and keep on doing so until all the files from the list have been "processed".
My code so far is as follows.
#!/bin/bash
number_of_files_in_folder=$(cat list.txt | wc -l)
max_number_of_files_to_process=4
Translated_files=/home/german_translated_files/
while IFS= read -r files
do
while [[ $number_of_files_in_folder -gt 0 ]]; do
i=1
while [[ $i -le $max_number_of_files_to_process ]]; do
my_first_function "$files" & # I execute my translation function for each file, as it can only perform 1 file per execution
find /home/german_translator/ -name '*.logs' -exec mv {} $Translated_files \; # As there will be several files generated, I have them copied to another folder
sed -i "/$files/d" list.txt # We remove the processed file from within our list.txt file.
my_second_function # Without parameters as it will process all the files copied at step 2.
done
# here, I want to have all the files processed and don't stop after the first iteration
done
done < list.txt
Unfortunately, as I am not quite good at shell scripting, I do not know how to structure it so that it won't waste any resources and mostly, to make sure that it "processes" everything from that file.
Do you have any advice on how to achieve what I am trying to achieve?
only 4 items out of the file. Once the functions have been executed for these 4 files, go over to the next 4
Seems to be quite easy with xargs.
your_function() {
echo "Do something with $1 $2 $3 $4"
}
export -f your_function
xargs -d '\n' -n 4 bash -c 'your_function "$#"' _ < list.txt
xargs -d '\n' for each line
-n 4 take for arguments
bash .... - run this command with 4 arguments
_ - the syntax is bash -c <script> $0 $1 $2 etc..., see man bash.
"$#" - forward arguments
export -f your_function - export your function to environment so child bash can pick it up.
I execute my translation function for each file
So you execute your translation function for each file, not for each 4 files. If the "translation function" is really for each file with no inter-file state, consider rather executing 4 processes in parallel with same code and just xargs -P 4.
If you have GNU Parallel it looks something like this:
doit() {
my_first_function "$1"
my_first_function "$2"
my_first_function "$3"
my_first_function "$4"
my_second_function "$1" "$2" "$3" "$4"
}
export -f doit
cat list.txt | parallel -n4 doit
I wrote a small script which:
prints the content of a file (generated by another application) on paper with a matrix printer
prints the same line into a backup file
removes the original file.
The script runs every minute by a cronjob and works fine as long as there are files to print. If there are no files to print, it prints an empty line on the matrix printer and in the backup file. I don't understand why this happens as i implemented an if statement which checks if there is a file to print before the print command is executed. This behaviour only happens if the script is executed by the cron and not if i execute it manually with ./script.sh. What's the reason of this? and how can i solve it?
Something i noticed on the side is that if I place an echo "hi" command in the script, its printed to the matrix printer and the backup file. I expected that its printed to the console console when it has no >> something behind. How does this work?
The script:
#!/bin/bash
# Make sure the backup directory exists
if [ ! -d /home/user/backup_logprint ]
then
mkdir /home/user/backup_logprint
fi
# Print the records if there are any
date=`date +%Y-%m-%d`
filename='_logprint_backup'
printer_path="/dev/usb/lp0"
if [ `ls /tmp/ | grep logprint | wc -l` -gt 0 ]
then
for f in `ls /tmp | grep logprint`
do
echo `cat /tmp/$f` >> "/home/user/backup_logprint/$date$filename"
echo `cat /tmp/$f` >> $printer_path
rm "/tmp/$f"
done
fi
There's no need for ls or an if statement. Just use a proper glob in the for loop, and if no file match, the loop won't be entered.
#!/bin/bash
# Don't check first; just let mkdir decide if
# anything actually needs to be created.
d=/home/user/backup_logprint
mkdir -p "$d"
filename=$(date +"$d/%Y-%m-%d_logprint_backup")
printer_path="/dev/usb/lp0"
# Cause non-matching globs to expand to an empty
# sequence instead of being treated literally.
shopt -s nullglob
for f in /tmp/*logprint*; do
cat "$f" > "$printer_path" && mv "$f" "$d"
done
I never used shell script, but now I have to , here is what I'm trying to do :
#!/bin/bash
echo running the program
./first
var = ($(ls FODLDER |wc -l)) #check how many files the folder contains
echo $var
if( ["$var" -gt "2"] #check if there are more the 2file
then ./second
fi
the scriopt crashes at the if statement. how may I solve this
Many:
var = ($(ls FODLDER |wc -l))
This is wrong, you cannot have those spaces around =.
if( ["$var" -gt "2"]
Your ( is not doing anything there, so it has to be deleted. Also, you need spaces around [ and ].
All together, this would make more sense:
#!/bin/bash
echo "running the program"
./first
var=$(find FOLDER -maxdepth 1 -type f|wc -l) # better find than ls
echo "$var"
if [ "$var" -gt "2" ]; then
./second
fi
Note:
quote whenever you echo, specially when handling variables.
see another way to look for files in a given path. Parsing ls is kind of evil.
indent your code for better readibility.
Edit your script.bash file as follow:
#!/bin/env bash
dir="$1"
echo "running the program"
./first
dir_list=( $dir/* ) # list files in directory
echo ${#dir_list[#]} # count files in array
if (( ${#dir_list[#]} > 2 )); then # test how many files
./second
fi
Usage
script.bash /tmp/
Explaination
You need to learn bash to avoid dangerous actions!
pass the directory to work with as first argument in the command line (/tmp/ → `$1)
use glob to create an array (dir_list) containing all file in given directory
count items in array (${#dir_list[#]})
test the number of item using arithmetic context.
I'm completing the following for one of my assignments using Korn shell.
For each argument in the argument list (which becomes the current pathname):
Check whether the current pathname is a directory, and if so:
Initialize a variable maxsubdir with the null (empty) string, and
a maxentries variable to 0;
For each entry in the directory check if that entry represents a
directory and if so, find the numbers of entries in that
subdirectory with a pipe consisting of ls -l and wc, and save the
result in a variable named curentries.
Compare curentries with maxentries, and if curentries is greater,
update maxsubdir and maxentries. (--10 points)
When the for cycle for a directory is completed, display (with
echo) the directory name, maxsubdir and maxentries (with appropriate
explanatory text.)
If the pathname in a) is not a directory, display the pathname
and an explanatory text saying that the pathname does not represent
a directory.
Go to the next command line argument (pathname) and repeat 1-7
The execution of the script ends when all pathnames are processed (the while is completed )
This is the code I have for it so far (EDITED):
#!/bin/ksh
directoy=$1
while [ $# -ne 0 ]; do
if [ -d $1 ]; then
maxsubdir=
maxentries=0
for x in $1; do
echo "Checking if $1 represents a directory..\n"
curentries="ls -l | wc"
if [ $curentries > $maxentries ]; then
maxentries=$curentries
maxsubdir=$curentries
fi;
done
echo "The directory structure of $1 is … \n"
echo "Maximum sub directories: \n"
echo "$maxsubdir\n"
echo "Maximum directory entries: \n"
echo "$maxentries"
fi
done
Where do I need to insert the "shift" command since I Unix can only handle a limited number of arguments?
Is my syntax appropriate? Or do I have syntax errors on sort lines?
Script seems to run but does not produce output to screen? Perhaps it's endless?
Have a look here and see if this helps out. Explanations are in the code.
#!/bin/ksh
directory=$1
# check whether the entered path is a directory
if [ -d $1 ];then # yes, it's a directory
maxsubdir=null
maxentries=0
echo "$1 is a directory"
# you are only counting lines, add -l to wc
# also you have to not count the first line. it's returns the size
curentries=`ls -l $1 | wc -l`
echo ${curentries}
fi
You don't.
You do have some errors.
Or perhaps, it never reaches that code?
Your assignment says specifically to use a for loop, and you've implemented a while loop.
I'll get you started:
for directory in $*; do
cd "$directory"
curentries=$(ls -1 | wc -l)
for entry in $(ls -1); do
...
done
done
Two questions: how can I write a shell variable from this script into its child script?
Are there any easier ways to do this?
If you can't follow what I'm doing, I'm:
1) starting with a list of directories whose names will be stored as values taken by $i
2) cd'ing to every value of $i and ls'ing its contents
3) echoing its contents into a new script with the name of the directory via cat
4) using echo and cat to write a new script that contains the ls'd values of $i and sends them all to a blogging email address called $i#tumblr.com
#/bin/sh
read -d '' commands <<EOF
#list of directories goes here
dir1
dir2
dir3
etc...
EOF
for i in $commands
do
cd $SPECIALPATH/$i
echo ("#/bin/sh \n read -d '' directives <<EOF \n") | cat >> $i.sh
ls | cat >> $i.sh
echo ("EOF \n for q in $directives \n do \n uuencode $q $q | sendmail $i \n done \n") | cat >> $i.sh
# NB -- I am asking the script to write the shell variable $i into the new
# script, called $i.sh, as the email address specified, in the middle of an
# echo statement... I am well aware that it doesn't work as is
chmod +x $i.sh
./$i.sh
done
You are abusing felines a lot - you should simply redirect, rather than pipe to cat which appends.
You can avoid the intermediary $i.sh file by bundling all the output that goes to the file with a single I/O redirection that pipes direct into a shell - no need for the intermediate file to clean up (you didn't show that happening) or the chmod operation.
I would have done this using braces:
{
echo "..."
ls
echo "..."
} | sh
However, when I looked at the script in that form, I realized that wasn't necessary. I've left the initial part of your script unchanged, but the loop is vastly simpler like this:
#/bin/sh
read -d '' commands <<EOF
#list of directories goes here
dir1
dir2
dir3
etc...
EOF
for i in $commands
do
(
cd $SPECIALPATH/$i
ls |
while read q
do uuencode $q $q | sendmail $i
done
)
done
I'm assuming the sendmail command works - it isn't the way I'd try sending email. I'd probably use mailx or something similar, and I'd avoid using uuencode too (I'd use a base-64 encoding, left to my own devices):
do uuencode $q $q | mailx -s "File $q" $i#tumblr.com
The script also uses parentheses around the cd command. It means that the cd command and what follows is run in a sub-shell, so the parent script does not change directory. In this case, with an absolute pathname for $SPECIALDIR, it would not matter much. But as a general rule, it often makes life easier if you isolate directory changes like that.
I'd probably simplify it still further for general reuse (though I'd need to add something to ensure that SPECIALPATH is set appropriately):
#/bin/sh
for i in "$#"
do
(
cd $SPECIALPATH/$i
ls |
while read q
do uuencode $q $q | sendmail $i
done
)
done
I can then invoke it with:
script-name $(<list-of-dirs)
That means that without editing the script, it can be reused for any list of directories.
Intermediate step 1:
for i in $commands
do
(
cd $SPECIALPATH/$i
{
echo "read -d '' directives <<EOF"
ls
echo "EOF"
echo "for q in $directives"
echo "do"
echo " uuencode $q $q | sendmail $i"
echo "done"
} |
sh
)
done
Personally, I find it easier to read the generated script if the code that generates makes the generated script clear - using multiple echo commands. This includes indenting the code.
Intermediate Step 2:
for i in $commands
do
(
cd $SPECIALPATH/$i
{
ls |
echo "while read q"
echo "do"
echo " uuencode $q $q | sendmail $i"
echo "done"
} |
sh
)
done
I don't need to read the data into a variable in order to step through each item in the list once - simply read each line in turn. The while read mechanism is often useful for splitting up a line into multiple variables too: while read var1 var2 var3 junk will read the first field into $var1, the second into $var2, the third into $var3, and if there's anything left over, it goes into $junk. If you've generated the data accurately, there won't be any junk; but sometimes you have to deal with other people's data.
If the generated script is meant to be temporary, I would not use files. Besides, chmoding them to executable sounds unsafe. When I needed to parallel my scripting, I used a bash script to form a set of commands (in an array, split the array in two, then implode the array) to a single \n-separated string and then pass that to a new bash instance.
Basically, in bash:
for orig in "$#"
do
commands="$commands echo \"echoeing stuff here for arguments $orig\" \n"
done
echo -e $commands |bash
And a small tip: if the script doesn't need supervising, throw in a & after the piped bash to make your first script quit and do the rest of the work forked background.
If you export a variable
export VAR1=FOO
it'll be present in any child processes.
If you take a look at the init scripts, /etc/init..d/* you'll notice that many source another file full of "external" definitions. You could set up a file like that and have your child script source these files.