How to log the compressed zip files when using a function in shell script - linux

I am working on a shell script which should validate .gz files in multiple folders in linux and then gzip them if a particular file not zipped and if already the file is zipped, purge them with following condition.
a) All these files in folders have *.log..gz as extension
So i was using functions and find cmd to achieve the same.
Script seems to be working fine but its not logging the zipped files information to log file, however its spooling about already zipped files in the folder to log. is this the correct way using functions?
#!/bin/bash
DIR_PATH="/var/log"
LOG="/tmp/test.log"
VARLOG_PATH=("$DIR_PATH"{"Kevin","John","Robin","Pavan"})
fun_zip_log(){
for i in `find "$i" -type f \( -name "*.log.20*" 2>/dev/null \) `; do
gzip "$i" ; done >> $LOG
}
fun_purge_log(){
for i in `find "$i" -type f \( -name "log.20*" 2>/dev/null \) `; do rm -f
"$i" ; done >> $LOG
}
validate_zip(){
for file in $i/*.gz
do
if ! [ -f "$file" ];
then
echo "$file is getting zipped" >> $LOG
fun_zip_log "$i"
else
echo "$file is already zipped" >> $LOG
fun_purge_log "$i"
fi
done
}
#MainBlock
for i in "${VARLOG_PATH[#]}"
do
if [ -d "$i" ] && [ "$(ls -A "$i" |wc -l )" -gt 0 ]; then
echo "Searching for files in directory : "$i" " >> $LOG
validate_zip "$i"
else
echo "No files exist in directory : "$i" " >> $LOG
fi
done
exit
####LOG FILE###
Searching for files in directory : /var/log/Kevin
[*.gz] is getting zipped.
Searching for files in directory : /var/log/John
/var/log/John/instrumentation.log.2018-06-20.gz is already zipped
/var/log/John/instrumentation.log.2018-06-21.gz is already zipped
No files exist in directory : /var/log/Robin
Searching for files in directory : /var/log/Pavan
[*.gz] is getting zipped.

Your code is very muddled and confusing. For example in this:
fun_purge_log(){
for i in `find "$i" -type f \( -name "log.20*" 2>/dev/null \) `; do rm -f
"$i" ; done >> $LOG
}
for file in $i/*.gz
do
...
fun_purge_log "$i"
In the calling code you're looping setting a variable file but then passing the directory "$i" to your function to then try to find the files again.
Within fun_purge_log() you're ignoring the argument being passed in and then using the global variable i as both the directory argument to find and also to loop through the list of files output by find - why not pick a new variable name and use some local variables?
I can't imagine what you think 2>/dev/null is going to do in \( -name "*log.20*" 2>/dev/null \).
You're trying to append something to $LOG but you aren't printing anything to append to it.
Run your code through shellcheck (e.g. at shellcheck.net), read http://mywiki.wooledge.org/BashFAQ/001, https://mywiki.wooledge.org/Quotes and https://mywiki.wooledge.org/ParsingLs, and really just THINK about what each line of your code is doing. Correct the issues yourself and then let us know if you still have a problem. Oh, and by convention and to avoid clashing with other variables don't use all capitals for non-exported variable names and lastly use $(command) instead of `command`.

Related

Script to rename files in subfolders to different name

I have a directory with several folders inside. Inside the folders I have a worksheet with the same name in all folders. I need to run a script that randomly changes the name of the worksheets so that I can throw them all in the same folder. For example: worksheet1
worksheet2
worksheet3.
today they are all called spreadsheet.csv
I have a sketch in linux.
need help please.
NN=0;
for arq in $(ls -1 *.csv);do
let NN++;
rename -n 's/'${arq}'/spreadsheet'${NN}'.csv/' ${arq};
done
(this search all files csv, but dont is recursive. Work only in one directory)
Use globstar:
n=0
shopt -s globstar nullglob || exit
for f in **/*.csv; do
until dst=spreadsheet$(( ++n )).csv; [[ ! -e ${dst} ]]; do
continue
done
mv -i -- "${f}" "${dst}"
done
Sample files:
$ find . -name "*.csv"
./spreadsheet.csv
./sub1/spreadsheet.csv
./sub1/sub2/spreadsheet.csv
One idea:
n=0
while read -r oldname
do
((++n))
newname="${oldname##*/}"
newname="${newname//.csv/-$n.csv}"
echo mv "$oldname" "$newname"
done < <(find . -name spreadsheet.csv)
This generates:
mv ./spreadsheet.csv spreadsheet-1.csv
mv ./sub1/spreadsheet.csv spreadsheet-2.csv
mv ./sub1/sub2/spreadsheet.csv spreadsheet-3.csv
Once OP is satisifed with the output, remove the echo and run the script again.
After removing the echo and running again:
$ find . -name "*.csv"
./spreadsheet-1.csv
./spreadsheet-2.csv
./spreadsheet-3.csv

linux: how to batch rename folder name and the file name under folder

I need a help to finish a script to rename a folders and .
eg: my current folders and files like below:
Gideon/gideon_lisha/Gideon_samuel/Gideon_nathan.xml
Gideon/lisha_gideon/Gideon_noah.xml
...
I want a shell command to rename them like below:
Liang/Liang_lisha/Liang_samuel/Liang_nathan.xml
Liang/lisha_Liang/Liang_noah.xml
...
I tied:
#!/bin/bash
path=$1
filename=$2
newfilename=$3
echo "We are finding '$filename' under the folder '$path'"
count=1
for i in `find $path -iname *$filename*`
do
newpath=`echo $i | sed "s/$filename/$newfilename/g"`
sudo mv "$i" "$newpath"
echo "${count}: Renaming $i to $newpath"
let count++
done
but the script will stop to:
Liang/gideon_lisha/Gideon_samuel/Gideon_nathan.xml
because it changed the folder name, so that can not find the next path. I do not know how let the script run from inner to outer instead of running outer to inner.
finally, I found out the anwser:
#!/bin/bash
path=$1
filename=$2
newfilename=$3
echo "We are finding '$filename' under the folder '$path'"
count=1
for i in `find $path -iname "*$filename*" | tac`
do
newpath=`echo $i | sed "s#\(.*\)$filename#\1$newfilename#i"`
sudo mv "$i" "$newpath"
echo "${count}: Renaming $i to $newpath"
let count++
done
really thank #susbarbatus !

Verify if a file was created shell

I have to write a shell script that will monitor some folders given in the command and give a message if a certain file will be created inside them (the name of the file will be read from keyboard).
Can anyone tell me why is this not working?
#!/bin/sh
f=`read filename`
isIn=0
for dir in $*
do
if [ ! -d $dir ]
then
echo $dir is not a directory.
fi
for i in `find $dir -type f`
do
if [ $f=$i ]
then
echo The file $f already exists.
isIn=1
break
fi
done
if [ $isIn -eq 0 ]
then
sleep 20
isIn=0
for i in `find $dir -type f`
do
if [ $f=$i ]
then
echo The file was created\!
isIn=1
break
fi
done
fi
if [ $isIn -eq 0 ]
then
echo The file was not created\!
fi
done
The idea i used was that i take all the files from the directory and verify is the file isn't already there.
If it is - show message and move to the next directory.
If not then I 'wait'. if in the time i waited that certain file was created, it would have appeared in the list of all files, and i check for it.
My problem is that no matter what file I read from the keyboard, i would get the message "The file already exists." without telling me the name of the file.
Replace f=`read filename` with the correct usage read f or read -p filename: f.
find $dir -type f prints the full file name including the directory path. Since you want just the basename, replace both lines
for i in `find $dir -type f`
with
for i in `find $dir -type f -printf '%f\n'`
Each operator and operand in [ ] must be a separate argument. Thus replace both lines
if [ $f=$i ]
with
if [ "$f" = $i ]

Rewrite a script so it takes option arguments to control its behaviour

I created a script and it moves files with different extensions to their specified directories.
If the directory is not there, it creates another one (where the files will go), and it creates another directory where the remaining files with different extensions will go.
My first problem is that I want when I put -d and full path on the terminal it should move only media files, -l and full path to move all text files, then -x to change the extension to uppercase, then -u to lowercase.
Can somebody modify it for me and show me how to overcome this problem?
#!/bin/bash
From="/home/elg19/lone/doc"
To="/home/elg19/mu"
WA="/home/elg19/du"
MA="/home/elg19/dq"
WQ="/home/elg19/d2"
# this function checks if the directory exits and creates one if it does not then moves all doc files
function mama(){
if [[ ! -d "$WA" ]]; then
mkdir -p "$WA"
fi
cd "$From"
for i in pdf txt doc; do
find . -type f -name "*.${i}" -exec mv "{}" "$WA" \;
done
}
# this function checks if the directory exits and creates one if it does not then moves all media files
function so(){
if [[ ! -d "$To" ]]; then
mkdir -p "$To"
fi
cd "$From"
for i in mp3 mp4 swf; do
find . -type f -name "*.${i}" -exec mv "{}" "$To" \;
done
}
# this function checks if the directory exits and creates one if it does not then moves all image files
function soa(){
if [[ ! -d "$MA" ]]; then
mkdir -p "$MA"
fi
cd "$From"
for i in jpg gif png; do
find . -type f -name "*.${i}" -exec mv "{}" "$MA" \;
done
}
# this function checks if the directory exits and creates one if it does not then moves all the remaining files
function soaq(){
if [[ ! -d "$WQ" ]]; then
mkdir -p "$WQ"
fi
cd "$From"
for i in *; do
find . -type f -name "*.${i}" -exec mv "{}" "$WQ" \;
done
}
mama
so
soa
soaq
I don't know if the options suggested are mnemonic in your native language, but they are counter-mnemonic in English. I would suggest something more like:
-m path Move media files
-t path Move text files
-u Change extensions to upper-case
-l Change extensions to lower-case
The command to use for regular argument parsing like this is getopts (plural - many systems also have a command getopt, singular, which has different characteristics altogether).
The referenced page gives an example of how to use it:
The following example script parses and displays its arguments:
aflag=
bflag=
while getopts ab: name
do
case $name in
a) aflag=1;;
b) bflag=1
bval="$OPTARG";;
?) printf "Usage: %s: [-a] [-b value] args\n" $0
exit 2;;
esac
done
if [ ! -z "$aflag" ]; then
printf "Option -a specified\n"
fi
if [ ! -z "$bflag" ]; then
printf 'Option -b "%s" specified\n' "$bval"
fi
shift $(($OPTIND - 1))
printf "Remaining arguments are: %s\n" "$*"
The option -a doesn't take an argument; the option -b requires an argument.

Appending rather than overwriting files when moving

I have the following directory structure:
+-archive
+-a
+-data.txt
+-b
+-data.txt
+-incoming
+-a
+-data.txt
+-c
+-data.txt
How do I do the equivalent of mv incoming/* archive/ but have the contents of the files in incoming appended to those in archive rather than overwrite them?
# move to incoming/ so that we don't
# need to strip a path prefix
cd incoming
# create directories that are missing in archive
for d in `find . -type d`; do
if [ ! -d "../archive/$d" ]; then
mkdir -p "../archive/$d"
fi
done
# concatenate all files to already existing
# ones (or automatically create them)
for f in `find . -type f`; do
cat "$f" >> "../archive/$f"
done
This should find any file in incoming and concatenate it to an existing file in archive.
The important part is to be inside incoming, because else we'd had to strip the path prefix (which is possible, but in the above case unnecessary). In the above case, a value of $f typically looks like ./a/data.txt, and hence the redirection goes to ../archive/./a/data.txt.
run it on the current directory.
find ./incoming -type f | while read -r FILE
do
dest=${FILE/incoming/archive}
cat "$FILE" >> "$dest"
done
the one in incoming/c would not be appended though
Here's a version with proper quoting:
#!/bin/sh
if [ -z "$1" ]; then
# acting as parent script
find incoming -type f -exec "$0" {} \;
else
# acting as child script
for in_file; do
if [ -f "$in_file" ]; then
destfile="${in_file/incoming/archive}"
test -d "$(dirname "$destfile")" || mkdir -p "$_"
cat "$in_file" >> "$destfile" &&
rm -f "$in_file"
fi
done
fi

Resources