I need to extract entries from a log file and put them on an errors file.
I don't want to duplicate the entries on the errors file every time that the script is run, so I create this:
grep $1 $2 | while read -r line ; do
echo "$line"
if [ ! -z "$line" ]
then
echo "Line is NOT empty"
if grep -q "$line" $3; then
echo "Line NOT added"
else
echo $line >> $3
echo "Line added"
fi
fi
done
And is run using:
./log_monitor.sh ERROR logfile.log errors.txt
The first time that the script runs it finds the entries, and create the errors file (there are no errors file before).
The next time, this line never found the recently added lines to the errors file,
if grep -q "$line" $3;
therefore, the script adds the same entries to the errors file.
Any ideas of why this is happening?
This most likely happens because you are not searching for the line itself, but treating the line as regex. Let's say you have this file:
$ cat file
[ERROR] This is a test
O This is a test
and you try to find the first line:
$ grep "[ERROR] This is a test" file
O This is a test
As you can see, it does not match the line we're looking for (causing duplicates) and does match a different line (causing dropped entries). You can instead use -F -x to search for literal strings matching the full line:
$ grep -F -x "[ERROR] This is a test" file
[ERROR] This is a test
Applying this to your script:
grep $1 $2 | while read -r line ; do
echo "$line"
if [ ! -z "$line" ]
then
echo "Line is NOT empty"
if grep -F -x -q "$line" $3; then
echo "Line NOT added"
else
echo $line >> $3
echo "Line added"
fi
fi
done
And here with additional fixes and cleanup:
grep -e "$1" -- "$2" | while IFS= read -r line ; do
printf '%s\n' "$line"
if [ "$line" ]
then
echo "Line is NOT empty"
if grep -Fxq -e "$line" -- "$3"; then
echo "Line NOT added"
else
printf '%s\n' "$line" >> "$3"
echo "Line added"
fi
fi
done
PS: this could be shorter, faster and have a better time complexity with a snippet of awk.
There's no need to check for blank lines; the first grep only checks lines with the word "ERROR", which cannot be blank.
If you can do without the diagnostic echo messages, pretty much the whole of that code boils down what might be done using two greps, a bash process substitution, sponge, and touch for the first-run case:
[ ! -f $3 ] && touch $3 ; grep -vf $3 <(grep $1 $2) | sponge -a $3
Related
I want to find a file. If user input "file.txt", then I need to echo the "file.txt". But if file name not found, just echo "not found". The code i have is for finding a word in a file. But how if I just need to search the file. If I write : result=`grep $filename`, it won't work. What's the format for grep filename?
#!/bin/bash
echo -n "File name : "
read filename
echo -n "Word : "
read word
result=`grep $word $filename`
if [ "$result" != "" ]
then
echo "$result"
else
echo "Not found"
fi
It looks like you are trying to determine if $word is present in $filename. If so:
if ! grep "$word" "$filename"; then
echo not found >&2
fi
The check to determine if the file exists seems redundant, since grep will emit an error message if the file does not exist, but perhaps you want something like:
if test -e "$filename" && ! grep "$word" "$filename"; then
echo "$word" is not found in "$filename" >&2
else
echo "$filename" does not exist >&2
fi
which will not print the redundant "not found" in addition to the error message from grep saying that the file does not exist.
With grep, you can try this way :
#!/bin/bash
OLDIFS="$IFS"
IFS=''
read -p "File name : " filename
IFS=$OLDIFS
grep -l '' "$filename" 2>/dev/null
! [ $? -eq 0 ] && echo "file $filename not found"
bemol : grep do not find a file with a size of 0.
If I read it right...
grep -l "$word" "$filename" 2>&- || echo "not found"
If $word exists in $filename the -l option will output the name of the file.
If not, grep will return an error exit code and the || ("or") will execute the echo.
$: echo foo > bar
$: grep -l foo ???
bar
$: rm bar
$: grep -l foo ??? 2>&- || echo "not found"
not found
The 2>&- closes STDERR so that grep doesn't throw an error message of its own.
Without it -
$: grep -l foo ??? || echo "not found"
grep: ???: No such file or directory
not found
Of course https://mywiki.wooledge.org/BashPitfalls#myprogram_2.3E.26- advises against closing standard streams, so you should probably use 2>/dev/null.
So, the whole script:
#!/bin/bash
read -p "Filename: " filename
read -p "Word: " word
grep -l "$word" "$filename" 2>/dev/null || echo "not found"
I am trying to check if a user types multiple arguments in a command line using case and if/else statements. What's wrong is that I keep getting the default case instead of the same command, but with 2 more arguments. For instance, one of my conditions is:
del)
if [ -z "$2" ] || [ -z "$3" ]
then
echo "Usage: removes a file"
else
echo "using Bash command: rm $2 $3"
rm $2 $3
echo done
fi
prints the first condition, but if I type, say, del aaa bbb, I get the default case, which is:
echo "ERROR: Unrecognized command"
I'm also using this to read a user's input, if that helps.
read -p "wcl> " -r wcl $2 $3
I don't really know if there's a better way to solve this without scrapping all my code and starting from scratch.
This is the full code:
#!/bin/bash
#use read command
echo Welcome to the Windows Command Line simulator!
echo Enter your commands below
while true
do
read -p "wcl> " -r wcl $2 $3
case $wcl in
dir)
echo "using Bash command: ls $2 $3"
ls
continue
;;
copy)
FILE="$2"
if [ "$#" -ne 3 ]
then
echo "Usage: copy sourcefile destinationfile"
else
echo "using Bash command: cp $2 $3"
if [ -f "$FILE" ]
then
cp $2 $3
else
echo "cannot stat $FILE: No such file or directory">&2
fi
echo done
fi
continue
;;
del)
if [ -z "$2" ] || [ -z "$3" ]
then
echo "Usage: removes a file"
else
echo "using Bash command: rm $2 $3"
rm $2 $3
echo done
fi
continue
;;
move)
if [ -z "$2" ] || [ -z "$3" ]
then
echo "Usage: moves a file to another file name and location"
else
echo "using Bash command: mv $2 $3"
mv $2 $3
echo done
fi
continue
;;
rename)
if [ -z "$2" ] || [ -z "$3" ]
then
echo "Usage: renames a file"
else
echo "using Bash command: mv $2 $3"
mv $2 $3
echo done
fi
continue
;;
ipconfig)
ifconfig eth0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1
continue
;;
exit)
echo "Goodbye"
exit 1
;;
^c)
echo "Goodbye"
exit 1
;;
*)
echo "ERROR: Unrecognized command"
continue
esac
done
You can't use read to set the positional parameters, although it isn't clear why you would need to here. Just use regular parameters.
while true
do
read -p "wcl> " -r wcl arg1 arg2
case $wcl in
dir)
echo "using Bash command: ls $arg1 $arg2"
ls "$arg1" "$arg2"
continue
;;
# ...
esac
done
The way read -r wcl $2 $3 is executed is that $2 and $3 are first expanded to give names that read will use to set variables. If those aren't set, then the command reduces to read -r wcl, and so your entire command line is assigned to the variable wcl, not just the command.
However, read by itself is not going to do the same parsing that the shell already does, if you goal is to write your own shell.
If you are really using bash, you can insert the words you read into positional parameters through an array. (You could also just leave them in the array, but the syntax for referring to positional parameters is simpler.)
# -a: read the successive words into an array
read -r -p "wcl> " -a params
# set the positional parameters to the expansion of the array
set -- "${params[#]}"
wcl=$1 # Or you could do the case on "$1"
This will also set $# to the number of words read, as a side-effect of setting the positional parameters.
As #chepner points outs, the read is problematic: It simply splits the input into whitespace-separated words, without respecting quotes, backslashes, and whatever other shell metacharacters you might want to implement. Doing a full bash-style parse of a command-line in bash itself would be quite a difficult exercise.
I'm writing a bash script to read a set of files line by line and perform some edits. To begin with, I'm simply trying to move the files to backup locations and write them out as-is, to test the script is working. However, it is failing to copy the last line of each file. Here is the snippet:
while IFS= read -r line
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
I obviously want to preserve whitespace when I copy the files, which is why I have set the IFS to null. I can see from the output that the last line of each file is being read, but it never appears in the output.
I've also tried an alternative variation, which does print the last line, but adds a newline to it:
while IFS= read -r line || [ -n "$line" ]
do
echo "Line is ***$line***"
echo "$line" >> $POM
done < $POM.backup
What is the best way to do this do this read-write operation, to write the files exactly as they are, with the correct whitespace and no newlines added?
The command that is adding the line feed (LF) is not the read command, but the echo command. read does not return the line with the delimiter still attached to it; rather, it strips the delimiter off (that is, it strips it off if it was present in the line, IOW, if it just read a complete line).
So, to solve the problem, you have to use echo -n to avoid adding back the delimiter, but only when you have an incomplete line.
Secondly, I've found that when providing read with a NAME (in your case line), it trims leading and trailing whitespace, which I don't think you want. But this can be solved by not providing a NAME at all, and using the default return variable REPLY, which will preserve all whitespace.
So, this should work:
#!/bin/bash
inFile=in;
outFile=out;
rm -f "$outFile";
rc=0;
while [[ $rc -eq 0 ]]; do
read -r;
rc=$?;
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
echo "$REPLY" >>"$outFile";
elif [[ -n "$REPLY" ]]; then ## incomplete line
echo "incomplete=\"$REPLY\"";
echo -n "$REPLY" >>"$outFile";
fi;
done <"$inFile";
exit 0;
Edit: Wow! Three excellent suggestions from Charles Duffy, here's an updated script:
#!/bin/bash
inFile=in;
outFile=out;
while { read -r; rc=$?; [[ $rc -eq 0 || -n "$REPLY" ]]; }; do
if [[ $rc -eq 0 ]]; then ## complete line
echo "complete=\"$REPLY\"";
printf '%s\n' "$REPLY" >&3;
else ## incomplete line
echo "incomplete=\"$REPLY\"";
printf '%s' "$REPLY" >&3;
fi;
done <"$inFile" 3>"$outFile";
exit 0;
After review i wonder if :
{
line=
while IFS= read -r line
do
echo "$line"
line=
done
echo -n "$line"
} <$INFILE >$OUTFILE
is juts not enough...
Here my initial proposal :
#!/bin/bash
INFILE=$1
if [[ -z $INFILE ]]
then
echo "[ERROR] missing input file" >&2
exit 2
fi
OUTFILE=$INFILE.processed
# a way to know if last line is complete or not :
lastline=$(tail -n 1 "$INFILE" | wc -l)
if [[ $lastline == 0 ]]
then
echo "[WARNING] last line is incomplete -" >&2
fi
# we add a newline ANYWAY if it was complete, end of file will be seen as ... empty.
echo | cat $INFILE - | {
first=1
while IFS= read -r line
do
if [[ $first == 1 ]]
then
echo "First Line is ***$line***" >&2
first=0
else
echo "Next Line is ***$line***" >&2
echo
fi
echo -n "$line"
done
} > $OUTFILE
if diff $OUTFILE $INFILE
then
echo "[OK]"
exit 0
else
echo "[KO] processed file differs from input"
exit 1
fi
Idea is to always add a newline at the end of file and to print newlines only BETWEEN lines that are read.
This should work for quite all text files given they are not containing 0 byte ie \0 character, in which case 0 char byte will be lost.
Initial test can be used to decided whether an incomplete text file is acceptable or not.
Add a new line if line is not a line. Like this:
while IFS= read -r line
do
echo "Line is ***$line***";
printf '%s' "$line" >&3;
if [[ ${line: -1} != '\n' ]]
then
printf '\n' >&3;
fi
done < $POM.backup 3>$POM
Can any one suggest me how to optimize below while loop which is part of a shell script.
function setvars() {
CONN_TSMP="$1"
USER="$2"
DB="$3"
IP="$4"
HOST="$5"
return
}
while read line; do
TST=`grep -w $line $FILE1`
ID=`echo $line | tr -d '\"'`
VARS=$(echo ${TST} | awk -F '"' '{print $2 " " $10 " " $22 " " $20 " " $18 }')
setvars $VARS
if [ -z "$IP" ]; then
IP=`echo "$HOST"`
fi
if [ "$USER" == "root" ] && [ -z $DB ]; then
TARGET=/home/database/data1/mysql_audit/sessions/root_sec
FILE=`echo "$ID-$CONN_TSMP-$USER#$IP.txt"`
else
TARGET=/home/database/data1/mysql_audit/sessions/user_sec
FILE=`echo "$ID-$CONN_TSMP-$USER#$IP.txt"`
fi
ls $TARGET/$FILE
if [ $? -ne 0 ]; then
echo -e "################################################################ \n" >> "$TARGET/$FILE"
echo "$TST" | awk -F 'STATUS="0"' '{print $2}'| sed "s/[</>]//g" >> "$TARGET/$FILE"
echo -e "\n" >> "$TARGET/$FILE"
fi
awk -F '"' '/"'$line'"/ {print "\n======================================\nTIMESTAMP=" $2 "\nSQLTEXT=" $10}' $FILE3 >> "$TARGET/$FILE"
done < "$FILE4"
According to my observation awk is taking more time.
Can any one help me how to write optimize the above code by replacing it with awk code (an awk while loop which replace entire while loop shown above) or by removing awk or sed or grep which take more time.
1) In setvars(), Remove the double quotes around the assignments. The double quotes force the shell to rescan the values. This is minor, but in large shell scripts, it can add up to quite a bit of processing time.
2) You have multiple VAR=echo $SOMEVAL. Just assign the value: ID=$HOST
FILE="$ID-$CONN_TSMP-$USER#$IP.txt"
3) You are running an external program 'ls' to check and see if a file exists. Instead, use the builtin shell commands: if [ ! -f "$TARGET/$FILE" ]; then ...; fi. If you want the output, just do an: echo "$TARGET/$FILE".
4) Open the output file once. This is much faster, but can make maintenance of the script quite difficult. Since you only have 4 echo lines, it may not help that much.
exec 4>>"$TARGET/$FILE"
if [ ! -f "$TARGET/$FILE" ]; then
echo -e ... >&4
...
fi
awk -f ... >&4
exec 4>&-
It's not possible to optimize your awk without seeing the data it is processing. You appear to have a more modern shell as there is a $(...) construct. Replace any backtick usage with $(...).
enter image description hereI was trying to see how a shell scripts work and how to run them, so I toke some sample code from a book I picked up from the library called "Wicked Cool Shell Scripts"
I re wrote the code verbatim, but I'm getting an error from Linux, which I compiled the code on saying:
'd.sh: line 3: syntax error near unexpected token `{
'd.sh: line 3:`gmk() {
Before this I had the curly bracket on the newline but I was still getting :
'd.sh: line 3: syntax error near unexpected token
'd.sh: line 3:`gmk()
#!/bin/sh
#format directory- outputs a formatted directory listing
gmk()
{
#Give input in Kb, output converted to Kb, Mb, or Gb for best output format
if [$1 -ge 1000000]; then
echo "$(scriptbc -p 2 $1/1000000)Gb"
elif [$1 - ge 1000]; then
echo "$$(scriptbc -p 2 $1/1000)Mb"
else
echo "${1}Kb"
fi
}
if [$# -gt 1] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [$# -eq 1] ; then
cd "$#"
fi
for file in *
do
if [-d "$file"] ; then
size = $(ls "$file"|wc -l|sed 's/[^[:digit:]]//g')
elif [$size -eq 1] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size ="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' |\
xargs -n 2 |\
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%39s %-39s\n", $1, $2}'
exit 0
if [$#-gt 1]; then
echo "Usage :$0 [dirname]" >&2; exit 1
elif [$# -eq 1]; then
cd "$#"
fi
for file in *
do
if [ -d "$file" ] ; then
size =$(ls "$file" | wc -l | sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size ="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(convert $size))|"
fi
done | \
sed 's/ /^^^/g' | \
xargs -n 2 | \
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%-39s %-39s\n", $1, $2 }'
exit 0
sh is very sensitive to spaces. In particular assignment (no spaces around =) and testing (must have spaces inside the [ ]).
This version runs, although fails on my machine due to the lack of scriptbc.
You put an elsif in a spot where it was supposed to be if.
Be careful of column alignment between starts and ends. If you mismatch them it will easily lead you astray in thinking about how this works.
Also, adding a set -x near the top of a script is a very good way of debugging what it is doing - it will cause the interpreter to output each line it is about to run before it does.
#!/bin/sh
#format directory- outputs a formatted directory listing
gmk()
{
#Give input in Kb, output converted to Kb, Mb, or Gb for best output format
if [ $1 -ge 1000000 ]; then
echo "$(scriptbc -p 2 $1/1000000)Gb"
elif [ $1 -ge 1000 ]; then
echo "$(scriptbc -p 2 $1/1000)Mb"
else
echo "${1}Kb"
fi
}
if [ $# -gt 1 ] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [ $# -eq 1 ] ; then
cd "$#"
fi
for file in *
do
if [ -d "$file" ] ; then
size=$(ls "$file"|wc -l|sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' |\
xargs -n 2 |\
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%39s %-39s\n", $1, $2}'
exit 0
By the way, with respect to the book telling you to modify your PATH variable, that's really a bad idea, depending on what exactly it advised you to do. Just to be clear, never add your current directory to the PATH variable unless you intend on making that directory a permanent location for all of your scripts etc. If you are making this a permanent location for your scripts, make sure you add the location to the END of your PATH variable, not the beginning, otherwise you are creating a major security problem.
Linux and Unix do not add your current location, commonly called your PWD, or present working directory, to the path because someone could create a script called 'ls', for example, which could run something malicious instead of the actual 'ls' command. The proper way to execute something in your PWD, is to prepend it with './' (e.g. ./my_new_script.sh). This basically indicates that you really do want to run something from your PWD. Think of it as telling the shell "right here". The '.' actually represents your current directory, in other words "here".