I am new to Shellscripting.I am working on a poc in which a script should read a log file and then append to a existing file for the purpose of alert.It should work as per below
There will be some predefined format according to which it will decide whether to append in file or not.For example:
WWXXX9999XS message
**XXX** - is a 3 letter acronym (application code) like for **tom** for tomcat application
9999 - is a 4 numeric digit in the range 1001-1999
**E or X** - For notification X ,If open/active alerts already existing for same error code and same message,new alerts will not be raised for existing one.Once you have closed existing alerts,it will raise alarm for new error.There is any change in message for same error code from existing one, it will raise a alarm even though open/active alerts present.
X option is only for drop duplicates on code and message otherwise all alert mechanisms are same.
**S** - is the severity level, I.e 2,3
**message** - is any text that will be displayed
The script will examine the log file, and look for error like cloud server is down,then it would append 'wwclo1002X2 cloud server is down'if its a new alert.
2.If the same alert is coming again,then it should append 'wwclo1002E2 cloud server is down
There are some very handy commands you can use to do this type of File manipulation. I've updated this in response to your comment to allow functionality that will check if the error has already been appended to the new file.
My suggestion would be that there is enough functionality here to warrant saving it in a bash script.
My approach would be to use a combination of less, grep and > to read and parse the file and then append to the new file. First save the following into a bash script (e.g. a file named script.sh)
#!/bin/bash
result=$(less $1 | grep $2)
exists=$(less $3 | grep $2)
if [[ "$exists" == "$result" ]]; then
echo "error, already present in file"
exit 1
else
echo $result >> $3
exit 0
fi
Then use this file in the command passing in the log file as the first argument, the string to search for as the second argument and the target results file as the third argument like this:
./script.sh <logFileName> "errorToSearchFor" <resultsTargetFileName>
Don't forget to run the file you will need to change the permissions - you can do this using:
chmod u+x script.sh
Just to clarify as you have mentioned you are new to scripting - the less command will output the entire file, the | command (an unnamed pipe) will pass this output to the grep command which will then search the file for the expression in quotes and return all lines from the file containing that expression. The output of the grep command is then appended to the new file with >>.
You may need to tailor the expression in quotes after grep to get exactly the output you want from the log file.
The filenames are just placeholders, be sure to update these with the correct file names. Hope this helps!
Note updated > to >> (single angle bracket overwrites, double angle bracket appends
I have a daily process which runs on Linux that returns a set of failed updated users, and need to delete these bad rows from the large user csv for importation into a database.
My output file contains the USER_ID for each failed user.
I'm trying to create an updated file with these removed.
I have reviewed the multitude of examples available, but none seem to work correctly. I've included a sample of the error file and the user file.
The first row is a header, and should be ignored
My error file:
"USER_ID"
"CA781558"
"LN764767"
My user file:
"USER_ID","FIRSTNAME","LASTNAME","LAST_ACTIVITY","GROUD_UID"
"CA781558","Dani","Roper","2015-07-17 19:47:21","CF93DF0A-BD23AF87D20A"
"BT055163","Alexis","Richardo","2016-04-19 21:23:08","CB71F91E-7E638292ABD5"
"LN764767","Peter","Rajosz","2016-03-18 11:59:29","973C4AD2-63BA12BB91CD"
"TN479717","Jerry","Alindos","2015-06-12 07:37:56","1DA745BA-71CB88AA91EA"
"FR915163","Alexis","Richardo","2016-04-19 21:23:08","DBA8B91E-7A6B8292ABD5"
"GB135767","Peter","Rajosz","2016-03-18 11:59:29","AE3C4AD2-63BA181B91CD"
"SG839717","Jerry","Alindos","2015-06-12 07:37:56","1BA746BA-71CB88AA91EA"
Expected Output:
"USER_ID","FIRSTNAME","LASTNAME","LAST_ACTIVITY","GROUD_UID"
"BT055163","Alexis","Richardo","2016-04-19 21:23:08","CB71F91E-7E638292ABD5"
"TN479717","Jerry","Alindos","2015-06-12 07:37:56","1DA745BA-71CB88AA91EA"
"FR915163","Alexis","Richardo","2016-04-19 21:23:08","DBA8B91E-7A6B8292ABD5"
"GB135767","Peter","Rajosz","2016-03-18 11:59:29","AE3C4AD2-63BA181B91CD"
"SG839717","Jerry","Alindos","2015-06-12 07:37:56","1BA746BA-71CB88AA91EA"
Can you help? Thank you in advance
You can use awk like this:
awk -F, 'FNR==NR{del[$1]; next} FNR==1 || !($1 in del)' err.txt file.txt
"USER_ID","FIRSTNAME","LASTNAME","LAST_ACTIVITY","GROUD_UID"
"BT055163","Alexis","Richardo","2016-04-19 21:23:08","CB71F91E-7E638292ABD5"
"TN479717","Jerry","Alindos","2015-06-12 07:37:56","1DA745BA-71CB88AA91EA"
"FR915163","Alexis","Richardo","2016-04-19 21:23:08","DBA8B91E-7A6B8292ABD5"
"GB135767","Peter","Rajosz","2016-03-18 11:59:29","AE3C4AD2-63BA181B91CD"
"SG839717","Jerry","Alindos","2015-06-12 07:37:56","1BA746BA-71CB88AA91EA"
I have a huge log files. Every time open the file will cause the system not responsive. I only need to check the log messages that contains certain strings.
Is there an simple way to do it?
$cat testlogfile.txt | grep --color=auto TRACE > newlogfile.txt
For example, your huge log file called testlogfile.txt. You only need check the log messages that contains "TRACE".
try this command under linux terminal and go to where the huge log is.
You can open the newlogfile.txt that only contains lines with "TRACE"
If you would like to exclude the lines with "TRACE", try -v option:
$cat testlogfile.txt | grep --color=auto -v TRACE > newlogfile.txt
I'm trying to extract a substring from an Oracle error message so I can email it off to an administrator using awk, this part of the code is trying to find where the important bit I want to extract.
starts here's what I have....
(The table name is incorrect to generate the error)
validate_iwpcount(){
DB_RETURN_VALUE=`sqlplus -s $DB_CRED <<END
SELECT count(COLUMN)
FROM INCORRECT_TABLE NAME;
exit
END`
a="$DB_RETURN_VALUE"
b="ERROR at line"
awk -v a="$a" -v b="$b" 'BEGIN{print index(a,b)}'
echo $DB_RETURN_VALUE
}
Strange thing is no matter how big that $DB_RETURN_VALUE is the return value from awk is always 28. Im assuming that somewhere in this error message there's something linux either thinks is a implcit delimiter of somesort and its messing with the count or something stranger. This works fine with regular strings as opposed to what oracle gives me.
Could anybody shine a light on this?
Many thanks
28 seems to be the right answer for the query you have (slightly amended to avoid an ORA-00936, and with tabs in the script). The message you're echoing includes a file expansion; the raw message is:
FROM IW_PRODUCTzS
*
ERROR at line 2:
ORA-00942: table or view does not exist
The * is expanded when you echo $DB_RETURN_VALUE, so the directory you're executing this from seem to have logs mail_files scripts in it, and they are being shown through expansion of the *. If you run it from different directories the echoed message length will vary, but the length of the actual message from Oracle stays the same - the length is changing (through the expansion) after the SQL*Plus call and after awk has done its thing. You can avoid that expansion with echo "$DB_RETURN_VALUE" instead, though I don't suppose you actually want to see that full message anyway in the end.
The substring from character 28 gives you what you want though:
validate_iwpcount(){
DB_RETURN_VALUE=`sqlplus -s $CENSYS_ORACLE_UID <<END
SELECT count(COLUMN_NAME)
FROM IW_PRODUCTzS;
exit
END`
# To see the original message; note the double-quotes
# echo "$DB_RETURN_VALUE"
a="$DB_RETURN_VALUE"
b="ERROR at line"
p=`awk -v a="$a" -v b="$b" 'BEGIN{print index(a,b)}'`
if [ ${p} -gt 0 ]; then
awk -v a="$a" -v p="$p" 'BEGIN{print substr(a,p)}'
fi
}
validate_iwpcount
... displays just:
ERROR at line 2:
ORA-00942: table or view does not exist
I'm sure that can be simplified, maybe into a single awk call, but I'm not that familiar with it.
Ok, here I'm again, struggling with ssh. I'm trying to retrieve some data from remote log file based on tokens. I'm trying to pass multiple tokens in egrep command via ssh:
IFS=$'\n'
commentsArray=($(ssh $sourceUser#$sourceHost "$(egrep "$v" /$INSTALL_DIR/$PROP_BUNDLE.log)"))
echo ${commentsArray[0]}
echo ${commentsArray[1]}
commax=${#commentsArray[#]}
echo $commax
where $v is something like below but it's length is dynamic. Meaning it can have many file names seperated by pipe.
UserComments/propagateBundle-2013-10-22--07:05:37.jar|UserComments/propagateBundle-2013-10-22--07:03:57.jar
The output which I get is:
oracle#172.18.12.42's password:
bash: UserComments/propagateBundle-2013-10-22--07:03:57.jar/New: No such file or directory
bash: line 1: UserComments/propagateBundle-2013-10-22--07:05:37.jar/nouserinput: No such file or directory
0
Thing worth noting is that my log file data has spaces in it. So, in the code piece I've given, the actual comments which I want to extract start after the jar file name like : UserComments/propagateBundle-2013-10-22--07:03:57.jar/
The actual comments are 'New Life Starts here' but the logs show that we are actually getting it till 'New' and then it breaks at space. I tried giving IFS but of no use. Probably I need to give it on remote but I don't know how should I do that.
Any help?
Your command is trying to run the egrep "$v" /$INSTALL_DIR/$PROP_BUNDLE.log on the local machine, and pass the result of that as the command to run via SSH.
I suspect that you meant for that command to be run on the remote machine. Remove the inner $() to get that to happen (and fix the quoting):
commentsArray=($(ssh $sourceUser#$sourceHost "egrep '$v' '/$INSTALL_DIR/$PROP_BUNDLE.log'"))
You should use fgrep to avoid regex special interpretation from your input:
commentsArray=($(ssh $sourceUser#$sourceHost "$(fgrep "$v" /$INSTALL_DIR/$PROP_BUNDLE.log)"))