Can any one suggest me how to optimize below while loop which is part of a shell script.
function setvars() {
CONN_TSMP="$1"
USER="$2"
DB="$3"
IP="$4"
HOST="$5"
return
}
while read line; do
TST=`grep -w $line $FILE1`
ID=`echo $line | tr -d '\"'`
VARS=$(echo ${TST} | awk -F '"' '{print $2 " " $10 " " $22 " " $20 " " $18 }')
setvars $VARS
if [ -z "$IP" ]; then
IP=`echo "$HOST"`
fi
if [ "$USER" == "root" ] && [ -z $DB ]; then
TARGET=/home/database/data1/mysql_audit/sessions/root_sec
FILE=`echo "$ID-$CONN_TSMP-$USER#$IP.txt"`
else
TARGET=/home/database/data1/mysql_audit/sessions/user_sec
FILE=`echo "$ID-$CONN_TSMP-$USER#$IP.txt"`
fi
ls $TARGET/$FILE
if [ $? -ne 0 ]; then
echo -e "################################################################ \n" >> "$TARGET/$FILE"
echo "$TST" | awk -F 'STATUS="0"' '{print $2}'| sed "s/[</>]//g" >> "$TARGET/$FILE"
echo -e "\n" >> "$TARGET/$FILE"
fi
awk -F '"' '/"'$line'"/ {print "\n======================================\nTIMESTAMP=" $2 "\nSQLTEXT=" $10}' $FILE3 >> "$TARGET/$FILE"
done < "$FILE4"
According to my observation awk is taking more time.
Can any one help me how to write optimize the above code by replacing it with awk code (an awk while loop which replace entire while loop shown above) or by removing awk or sed or grep which take more time.
1) In setvars(), Remove the double quotes around the assignments. The double quotes force the shell to rescan the values. This is minor, but in large shell scripts, it can add up to quite a bit of processing time.
2) You have multiple VAR=echo $SOMEVAL. Just assign the value: ID=$HOST
FILE="$ID-$CONN_TSMP-$USER#$IP.txt"
3) You are running an external program 'ls' to check and see if a file exists. Instead, use the builtin shell commands: if [ ! -f "$TARGET/$FILE" ]; then ...; fi. If you want the output, just do an: echo "$TARGET/$FILE".
4) Open the output file once. This is much faster, but can make maintenance of the script quite difficult. Since you only have 4 echo lines, it may not help that much.
exec 4>>"$TARGET/$FILE"
if [ ! -f "$TARGET/$FILE" ]; then
echo -e ... >&4
...
fi
awk -f ... >&4
exec 4>&-
It's not possible to optimize your awk without seeing the data it is processing. You appear to have a more modern shell as there is a $(...) construct. Replace any backtick usage with $(...).
Related
I need to extract entries from a log file and put them on an errors file.
I don't want to duplicate the entries on the errors file every time that the script is run, so I create this:
grep $1 $2 | while read -r line ; do
echo "$line"
if [ ! -z "$line" ]
then
echo "Line is NOT empty"
if grep -q "$line" $3; then
echo "Line NOT added"
else
echo $line >> $3
echo "Line added"
fi
fi
done
And is run using:
./log_monitor.sh ERROR logfile.log errors.txt
The first time that the script runs it finds the entries, and create the errors file (there are no errors file before).
The next time, this line never found the recently added lines to the errors file,
if grep -q "$line" $3;
therefore, the script adds the same entries to the errors file.
Any ideas of why this is happening?
This most likely happens because you are not searching for the line itself, but treating the line as regex. Let's say you have this file:
$ cat file
[ERROR] This is a test
O This is a test
and you try to find the first line:
$ grep "[ERROR] This is a test" file
O This is a test
As you can see, it does not match the line we're looking for (causing duplicates) and does match a different line (causing dropped entries). You can instead use -F -x to search for literal strings matching the full line:
$ grep -F -x "[ERROR] This is a test" file
[ERROR] This is a test
Applying this to your script:
grep $1 $2 | while read -r line ; do
echo "$line"
if [ ! -z "$line" ]
then
echo "Line is NOT empty"
if grep -F -x -q "$line" $3; then
echo "Line NOT added"
else
echo $line >> $3
echo "Line added"
fi
fi
done
And here with additional fixes and cleanup:
grep -e "$1" -- "$2" | while IFS= read -r line ; do
printf '%s\n' "$line"
if [ "$line" ]
then
echo "Line is NOT empty"
if grep -Fxq -e "$line" -- "$3"; then
echo "Line NOT added"
else
printf '%s\n' "$line" >> "$3"
echo "Line added"
fi
fi
done
PS: this could be shorter, faster and have a better time complexity with a snippet of awk.
There's no need to check for blank lines; the first grep only checks lines with the word "ERROR", which cannot be blank.
If you can do without the diagnostic echo messages, pretty much the whole of that code boils down what might be done using two greps, a bash process substitution, sponge, and touch for the first-run case:
[ ! -f $3 ] && touch $3 ; grep -vf $3 <(grep $1 $2) | sponge -a $3
I have created a bash script that is used to modify the ulimit of open files in the RHEL server.
so i have reading the lines in the file /etc/security/limits.conf and if the soft/hard limit of the open files are less than 10000 for '*' domain i am commenting the line and adding a new line with soft/hard limit as 10000.
The Script is working as designed but the sed command to comment a line in the script is not working.
Please find the full script below :-
#!/bin/sh
#This script would be called by '' to set ulimit values for open files in unix servers.
#
configfile=/etc/security/limits.conf
help(){
echo "usage: $0 <LimitValue>"
echo -e "where\t--LimitValue= No of files you want all the users to open"
exit 1
}
modifyulimit()
{
grep '*\s*hard\s*nofile\s*' $configfile | while read -r line ; do
firstChar="$(echo $line | xargs | cut -c1-1)"
if [ "$firstChar" != "#" ];then
hardValue="$(echo $line | rev | cut -d ' ' -f1 | rev)"
if [[ "$hardValue" -ge "$1" ]]; then
echo ""
else
sed -i -e 's/$line/#$line/g' $configfile
echo "* hard nofile $1" >> $configfile
fi
else
echo ""
fi
done
grep '*\s*soft\s*nofile\s*' $configfile | while read -r line ; do
firstChar="$(echo $line | xargs | cut -c1-1)"
if [ "$firstChar" != "#" ];then
hardValue="$(echo $line | rev | cut -d ' ' -f1 | rev)"
if [[ "$hardValue" -ge "$1" ]]; then
echo ""
else
sed -i -e 's/$line/#$line/g' $configfile
echo "* hard nofile $1" >> $configfile
fi
else
echo ""
fi
done
}
deleteEofTag(){
sed -i "/\b\(End of file\)\b/d" $configfile
}
addEofTag()
{
echo "#################End of file###################" >> $configfile
}
#-------------Execution of the script starts here ----------------------
if [ $# -ne 1 ];
then
help
else
modifyulimit $1
deleteEofTag
addEofTag
fi
The command sed -i -e 's/$line/#$line/g' $configfile when executed from the terminal is working absolutely fine and it is commenting the line but it is not working when i am executing it from the unix shell script.
interpolation does not work in single quote
use double quote and try
sed -i -e 's/$line/#$line/g'
sed -i -e "s/$line/#$line/g"
also you might try:
sed -i -e s/${line}/#${line}/g
as this will tell the script to take the value of the variable instead of variable as such.
I am having trouble with my newbie linux script which needs to count brackets and tell if they are matched.
#!/bin/bash
file="$1"
x="()(((a)(()))"
left=$(grep -o "(" <<<"$x" | wc -l)
rght=$(grep -o ")" <<<"$x" | wc -l)
echo "left = $left right = $rght"
if [ $left -gt $rght ]
then echo "not enough brackets"
elif [ $left -eq $rght ]
then echo "all brackets are fine"
else echo "too many"
fi
the problem here is i can't pass an argument through command line so that grep would work and count the brackets from the file. In the $x place I tried writing $file but it does not work
I am executing the script by writting: ./script.h test1.txt the file test1.txt is on the same folder as script.h
Any help in explaining how the parameter passing works would be great. Or maybe other way to do this script?
The construct <<< is used to transmit "the contents of a variable", It is not applicable to "contents of files". If you execute this snippet, you could see what I mean:
#!/bin/bash
file="()(((a)((a simple test)))"
echo "$( cat <<<"$file" )"
which is also equivalent to just echo "$file". That is, what is being sent to the console are the contents of the variable "file".
To get the "contents of a file" which name is inside a var called "file", then do:
#!/bin/bash
file="test1.txt"
echo "$( cat <"$file" )"
which is exactly equivalent to echo "$( <"$file" )", cat <"$file" or even <"$file" cat
You can use: grep -o "(" <"$file" or <"$file" grep -o "("
But grep could accept a file as a parameter, so this: grep -o "(" "$file" also works.
However, I believe that tr would be a better command, as this: <"$file" tr -cd "(".
It transforms the whole file into a sequence of "(" only, which will need a lot less to be transmitted (passed) to the wc command. Your script would become, then:
#!/bin/bash -
file="$1"
[[ $file ]] || exit 1 # make sure the var "file" is not empty.
[[ -r $file ]] || exit 2 # test if the file "file" exists.
left=$(<"$file" tr -cd "("| wc -c)
rght=$(<"$file" tr -cd ")"| wc -c)
echo "left = $left right = $rght"
# as $left and $rght are strictly numeric values, this integer tests work:
(( $left > $rght )) && echo "not enough right brackets"
(( $left == $rght )) && echo "all brackets are fine"
(( $left < $rght )) && echo "too many right brackets"
# added as per an additional request of the OP.
if [[ $(<"$file" tr -cd "()"|head -c1) = ")" ]]; then
echo "the first character is (incorrectly) a right bracket"
fi
enter image description hereI was trying to see how a shell scripts work and how to run them, so I toke some sample code from a book I picked up from the library called "Wicked Cool Shell Scripts"
I re wrote the code verbatim, but I'm getting an error from Linux, which I compiled the code on saying:
'd.sh: line 3: syntax error near unexpected token `{
'd.sh: line 3:`gmk() {
Before this I had the curly bracket on the newline but I was still getting :
'd.sh: line 3: syntax error near unexpected token
'd.sh: line 3:`gmk()
#!/bin/sh
#format directory- outputs a formatted directory listing
gmk()
{
#Give input in Kb, output converted to Kb, Mb, or Gb for best output format
if [$1 -ge 1000000]; then
echo "$(scriptbc -p 2 $1/1000000)Gb"
elif [$1 - ge 1000]; then
echo "$$(scriptbc -p 2 $1/1000)Mb"
else
echo "${1}Kb"
fi
}
if [$# -gt 1] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [$# -eq 1] ; then
cd "$#"
fi
for file in *
do
if [-d "$file"] ; then
size = $(ls "$file"|wc -l|sed 's/[^[:digit:]]//g')
elif [$size -eq 1] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size ="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' |\
xargs -n 2 |\
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%39s %-39s\n", $1, $2}'
exit 0
if [$#-gt 1]; then
echo "Usage :$0 [dirname]" >&2; exit 1
elif [$# -eq 1]; then
cd "$#"
fi
for file in *
do
if [ -d "$file" ] ; then
size =$(ls "$file" | wc -l | sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size ="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(convert $size))|"
fi
done | \
sed 's/ /^^^/g' | \
xargs -n 2 | \
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%-39s %-39s\n", $1, $2 }'
exit 0
sh is very sensitive to spaces. In particular assignment (no spaces around =) and testing (must have spaces inside the [ ]).
This version runs, although fails on my machine due to the lack of scriptbc.
You put an elsif in a spot where it was supposed to be if.
Be careful of column alignment between starts and ends. If you mismatch them it will easily lead you astray in thinking about how this works.
Also, adding a set -x near the top of a script is a very good way of debugging what it is doing - it will cause the interpreter to output each line it is about to run before it does.
#!/bin/sh
#format directory- outputs a formatted directory listing
gmk()
{
#Give input in Kb, output converted to Kb, Mb, or Gb for best output format
if [ $1 -ge 1000000 ]; then
echo "$(scriptbc -p 2 $1/1000000)Gb"
elif [ $1 -ge 1000 ]; then
echo "$(scriptbc -p 2 $1/1000)Mb"
else
echo "${1}Kb"
fi
}
if [ $# -gt 1 ] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [ $# -eq 1 ] ; then
cd "$#"
fi
for file in *
do
if [ -d "$file" ] ; then
size=$(ls "$file"|wc -l|sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' |\
xargs -n 2 |\
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%39s %-39s\n", $1, $2}'
exit 0
By the way, with respect to the book telling you to modify your PATH variable, that's really a bad idea, depending on what exactly it advised you to do. Just to be clear, never add your current directory to the PATH variable unless you intend on making that directory a permanent location for all of your scripts etc. If you are making this a permanent location for your scripts, make sure you add the location to the END of your PATH variable, not the beginning, otherwise you are creating a major security problem.
Linux and Unix do not add your current location, commonly called your PWD, or present working directory, to the path because someone could create a script called 'ls', for example, which could run something malicious instead of the actual 'ls' command. The proper way to execute something in your PWD, is to prepend it with './' (e.g. ./my_new_script.sh). This basically indicates that you really do want to run something from your PWD. Think of it as telling the shell "right here". The '.' actually represents your current directory, in other words "here".
For the first time the select display the options and it works fine but when I break from screenTwo function to come into this menu (previuos) it ask me for the selection but does not display the options, how can I handle this?
PS3="Please make a selection => " ; export PS3
select var in `awk -F "#" '{print $4}' ${Temp_Config_File} | uniq` PREVIOUS
do
echo "\n`date +"%d%m%Y%H%M%S"` Daemon $var selected " >> $Log_File
if [ -z $var ] ; then echo "\n\tPlease enter a valid choice \n " ; continue
else
if [ $var = "PREVIOUS" ]; then
#removeFiles
break
fi
screenTwo $var
fi
done
First Option : Please give me your idea.
Second option : Capture the return of screenTwo and whenever it is
break use awk to print the index with $4. (like below , but I dont
like it)
if [ $breakStat -eq 99 ]; then
i=1
echo "\n\nPlease choose one of the following deamon you wish to Start/Stop\n\n"
awk -F "#" '{print $4}' Temp_OPDaemon_Config.cfg | uniq | while read line
do
echo "${i}) ${line}"
let i=i+1
done
echo "${i}) PREVIOUS"
fi
Thanks
this is what I did to handle it .
PS3="Please make a selection => " ; export PS3
select var in `awk -F "#" '{print $4}' ${Temp_Config_File} | uniq` PREVIOUS
do
echo "${Date_Time} Daemon $var selected \n" >> $Log_File
if [ -z $var ] ; then echo "\n\tPlease enter a valid choice \n " ; continue
else
if [ $var = "PREVIOUS" ]; then
#removeFiles
break
fi
while :
do
screenTwo $var
breakStat=$?
if [ $breakStat -eq 99 ]; then
break
elif [ $breakStat -eq 98 ]; then
continue
fi
done
if [ $breakStat -eq 99 ]; then
echo "\n\nPlease choose one of the following deamon you wish to Start/Stop\n\n"
awk -F "#" '{print $4 }' ${Temp_Config_File} | uniq | awk -F "#" '{print NR ") " $0} END {print NR+1") PREVIOUS"}'
fi
fi
done