Experts I came again after reading how to provide minimal reproducible example, I am placing the question again.
I want to filter the fully qualified hostname(eg: dtc4028.ptc.db01.delta.com) and count the repetition on an individual host.
Below is my raw data:
Feb 24 07:20:56 dbv0102 postfix/smtpd[29531]: NOQUEUE: reject: RCPT from dtc4023.ptc.db01.delta.com[172.10.10.161]: 554 5.7.1 <beta_st#dtc.com>: Sender address rejected: Access denied; from=<beta_st#dtc.com> to=<stordb#dtc.com> proto=ESMTP helo=<dtc4023.ptc.db01.delta.com>
Feb 24 07:21:20 dbv0102 postfix/smtpd[29528]: NOQUEUE: reject: RCPT from dtc4023.ptc.db01.delta.com[172.10.10.161]: 554 5.7.1 <beta_st#dtc.com>: Sender address rejected: Access denied; from=<beta_st#dtc.com> to=<stordb#dtc.com> proto=ESMTP helo=<dtc4023.ptc.db01.delta.com>
Feb 21 05:05:06 dbv0102 postfix/smtpd[32001]: disconnect from dtc4028.ptc.db01.delta.com[172.12.78.81]
Feb 21 05:05:23 dbv0102 postfix/smtpd[32010]: connect from dtc4028.ptc.db01.delta.com[172.12.78.81]
Feb 21 05:06:15 dbv0102 postfix/smtpd[31994]: connect from dtc3024.ptc.db01.delta.com[172.10.10.166]
Feb 21 05:06:15 dbv0102 postfix/smtpd[31994]: disconnect from dtc3024.ptc.db01.delta.com[172.10.10.166]
Feb 21 13:05:08 dbv0102 postfix/smtpd[29043]: lost connection after CONNECT from dtc4028.ptc.db01.delta.com[172.12.78.81]
Feb 21 13:05:08 dbv0102 postfix/smtpd[29048]: lost connection after CONNECT from dtc4028.ptc.db01.delta.com[172.12.78.82]
What myself tried:
What I am doing here, Just taking desired column 1,2,4 and 8
$ awk '/from dtc/{print $1, $2, $4, $8}' maillog.log
Feb 24 dbv0102 RCPT
Feb 24 dbv0102 RCPT
Feb 21 dbv0102 dtc4028.ptc.db01.delta.com[172.12.78.81]
Feb 21 dbv0102 dtc4028.ptc.db01.delta.com[172.12.78.81]
Feb 21 dbv0102 dtc3024.ptc.db01.delta.com[172.10.10.166]
Feb 21 dbv0102 dtc3024.ptc.db01.delta.com[172.10.10.166]
Feb 21 dbv0102 after
Feb 21 dbv0102 after
Secondly, I am removing RCPT|after as these lines do not have hostnames and then also removing [] to just have hostname's and count their repition.
$ awk '/from dtc/{print $1, $2, $4, $8}' maillog.log| egrep -v "RCPT|after" | awk '{print $4}'| cut -d"[" -f1 | uniq -c
2 dtc4028.ptc.db01.delta.com
2 dtc3024.ptc.db01.delta.com
What I Wish:
I wish if this can be written more intelligently with the awk itself rather i'm doing it dirty way.
Note: Can we get only the FQDN hostnames like dtc4028.ptc.db01.delta.com after the 6th column.
Based on your shown samples, could you please try following. Written and tested in GNU awk.
awk '
match($0,/from .*com\[/){
count[substr($0,RSTART+5,RLENGTH-6)]++
}
END{
for(key in count){
print count[key],key
}
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
match($0,/from .*com\[/){ ##Using match function to match regex from .*com\[
count[substr($0,RSTART+5,RLENGTH-6)]++ ##Whenever match is having a regex matched so it sets RSTART and RLENGTH, RSTART tells us starting point of matched regex and RLENGTH is complete length.
}
END{ ##Starting END block of this program from here.
for(key in count){ ##Traversing through count array here.
print count[key],key ##Printing its key and value here.
}
}
' Input_file ##Mentioning Input_file name here.
$ awk -F'[[ ]' '$8=="from"{ cnt[$9]++ } END{ for (host in cnt) print cnt[host], host }' file
2 dtc4028.ptc.db01.delta.com
2 dtc3024.ptc.db01.delta.com
FROM:
noaccess 7491 6817 0 Jul 19 ? 281:19 /usr/java/bin/java -
server -Xmx128m -XX:+UseParallelGC -XX:ParallelGCThreads=4
oracle 1715 11577 0 Nov 23 ? 32:33
/appl/oracle/product/10.2.0/db_1/jdk/bin/java -server -Xmx256M -
XX:MaxPermSize=
virtuo 21844 21814 0 17:27:27 pts/3 0:00 grep java
TO:
noaccess 7491 6817 0 Jul 19 ? 281:19 /usr/java/bin/java -
server -Xmx128m -XX:+UseParallelGC -XX:ParallelGCThreads=4
oracle 1715 11577 0 Nov 23 ? 32:33
/appl/oracle/product/10.2.0/db_1/jdk/bin/java -server -Xmx256M -
XX:MaxPermSize=
I need to remove virtuo 21844 21814 0 17:27:27 pts/3 0:00 grep java line
I am really new to UNIX/LINUX command.
Assuming that your FROM output was generated from an 'X' command, so:
X | sed '$d'
That will remove the last line of the output. To remove a specific line, you could use something like:
X | sed '/regex/d'
That will remove every line that has a match on the passed regex.
I have a log directory that consists of bunch of log files, one log file is created once an system event has happened. I want to write an oneline bash script that always monitors the file list and display the content of the newly created file on the terminal. Here is what it looks like:
Currently, all I have is to display the content of the whole directory:
for f in *; do cat $f; done
It lacks the monitoring feature that I wanted. One limitation of my system is that I do not have watch command. I also don't have any package manager to install fancy tools. Raw BSD is all I have. I do have tail, I was thinking of something like tail -F $(ls) but this tails each file instead of the file list.
In summary, I want to modify my script such that I can monitor the content of all newly created files.
First approach - use a hidden file in you dir (in my example it has a name .watch). Then you one-liner might look like:
for f in $(find . -type f -newer .watch); do cat $f; done; touch .watch
Second approach - use inotify-tools: https://unix.stackexchange.com/questions/273556/when-a-particular-file-arrives-then-execute-a-procedure-using-shell-script/273563#273563
You can cram it into a one-liner if you want, but I'd recommend just running the script in the background:
#!/bin/bash
[ ! -d "$1" ] && {
printf "error: argument is not a valid directory to monitory.\n"
exit 1
}
while :; fname="$1/$(inotifywait -q -e modify -e create --format '%f' "$1")"; do
cat "$fname"
done
Which will watch the directory given as the first argument, and cat any new or changed file in that directory. Example:
$ bash watchdir.sh my_logdir &
Which will then cat new or changed files in my_logdir.
Using inotifywait in monitor mode
First this little demo:
Open one terminal and run this:
ext=(php css other)
while :;do
subname=''
((RANDOM%10))||printf -v subname -- "-%04x" $RANDOM
date >/tmp/test$subname.${ext[RANDOM%3]}
sleep 1
done
This will create randomly files named /tmp/test.php, /tmp/test.css and /tmp/test.other, but randomly (approx 1 time / 10), the name will be /tmp/test-XXXX.[css|php|other] where XXXX is an hexadecimal random number.
Open another terminal and run this:
waitPaths=(/{home,tmp})
while read file ;do
if [ "$file" ] &&
( [ -z "${file##*.php}" ] || [ -z "${file##*.css}" ] ) ;then
(($(stat -c %Y-%X $file)))||echo -n new
echo file: $file, content:
cat $file
fi
done < <(
inotifywait -qme close_write --format %w%f ${waitPaths[*]}
)
This may produce something like:
file: /tmp/test.css, content:
Tue Apr 26 18:53:19 CEST 2016
file: /tmp/test.php, content:
Tue Apr 26 18:53:21 CEST 2016
file: /tmp/test.php, content:
Tue Apr 26 18:53:23 CEST 2016
file: /tmp/test.css, content:
Tue Apr 26 18:53:25 CEST 2016
file: /tmp/test.php, content:
Tue Apr 26 18:53:27 CEST 2016
newfile: /tmp/test-420b.php, content:
Tue Apr 26 18:53:28 CEST 2016
file: /tmp/test.php, content:
Tue Apr 26 18:53:29 CEST 2016
file: /tmp/test.php, content:
Tue Apr 26 18:53:30 CEST 2016
file: /tmp/test.php, content:
Tue Apr 26 18:53:31 CEST 2016
Some explanation:
waitPaths=(/{home,tmp}) could be written waitPaths=(/home /tmp) or for only one directory: waitPaths=/var/log
if condition search for filenames matching *.php or *.css
(($(stat -c %Y-%X $file)))||echo -n new will compare creation and modification time.
inotifywait
-q to stay quiet (don't print more then required)
-m for monitor mode: Command don't termine, but print each matching event.
-e close_write react only to specified kind of event.
-f %w%f Output format: path/file
Another way:
There is a more sophisticated sample:
Listenning for two kind of events (CLOSE_WRITE | CREATE)
Using a list of new files flags for knowing which files are new when CLOSE_WRITE event occur.
In second console, hit Ctrl+C, or in new terminal, tris this:
waitPaths=(/{home,tmp})
declare -A newFiles
while read path event file; do
if [ "$file" ] && ( [ -z "${file##*.php}" ] || [ -z "${file##*.css}" ] ); then
if [ "$event" ] && [ -z "${event//*CREATE*}" ]; then
newFiles[$file]=1
else
if [ "${newFiles[$file]}" ]; then
unset newFiles[$file]
echo NewFile: $file, content:
sed 's/^/>+ /' $file
else
echo file: $file, content:
sed 's/^/> /' $path/$file
fi
fi
fi
done < <(inotifywait -qme close_write -e create ${waitPaths[*]})
May produce something like:
file: test.css, content:
> Tue Apr 26 22:16:02 CEST 2016
file: test.php, content:
> Tue Apr 26 22:16:03 CEST 2016
NewFile: test-349b.css, content:
>+ Tue Apr 26 22:16:05 CEST 2016
file: test.css, content:
> Tue Apr 26 22:16:08 CEST 2016
file: test.css, content:
> Tue Apr 26 22:16:10 CEST 2016
file: test.css, content:
> Tue Apr 26 22:16:13 CEST 2016
Watching for new files AND new lines in old files, using bash
There is another solution by using some bashisms like associative arrays:
Sample:
wpath=/var/log
while : ;do
while read -a crtfile ;do
if [ "${crtfile:0:1}" = "-" ] &&
[ "${crtfile[8]##*.}" != "gz" ] &&
[ "${files[${crtfile[8]}]:-0}" -lt ${crtfile[4]} ] ;then
printf "\e[47m## %-14s :- %(%a %d %b %y %T)T ##\e[0m\n" ${crtfile[8]} -1
tail -c +$[1+${files[${crtfile[8]}]:-0}] $wpath/${crtfile[8]}
files[${crtfile[8]}]=${crtfile[4]}
fi
done < <( /bin/ls -l $wpath )
sleep 1
done
This will dump each files (with filename not ending by .gz) in /var/log, and watch for modification or new files, then dump new lines.
Demo:
In a first terminal console, hit:
ext=(php css other)
( while :; do
subname=''
((RANDOM%10)) || printf -v subname -- "-%04x" $RANDOM
name=test$subname.${ext[RANDOM%3]}
printf "%-16s" $name
{
date +"%a %d %b %y %T" | tee /dev/fd/5
fortune /usr/share/games/fortunes/bofh-excuses
} >> /tmp/$name
sleep 1
done ) 5>&1
You need to have fortune installed with BOFH excuses librarie.
If you really not have fortune, you could use this instead:
LANG=C ext=(php css other)
( while :; do
subname=''
((RANDOM%10)) || printf -v subname -- "-%04x" $RANDOM
name=test$subname.${ext[RANDOM%3]}
printf "%-16s" $name
{
date +"%a %d %b %y %T" | tee /dev/fd/5
for ((1; RANDOM%5; 1))
do
printf -v str %$[RANDOM&12]s
str=${str// /blah, }
echo ${str%, }.
done
} >> /tmp/$name
sleep 1
done ) 5>&1
This may output something like:
test.css Thu 28 Apr 16 12:00:02
test.php Thu 28 Apr 16 12:00:03
test.other Thu 28 Apr 16 12:00:04
test.css Thu 28 Apr 16 12:00:05
test.css Thu 28 Apr 16 12:00:06
test.other Thu 28 Apr 16 12:00:07
test.php Thu 28 Apr 16 12:00:08
test.css Thu 28 Apr 16 12:00:09
test.other Thu 28 Apr 16 12:00:10
test.other Thu 28 Apr 16 12:00:11
test.php Thu 28 Apr 16 12:00:12
test.other Thu 28 Apr 16 12:00:13
In a second terminal console, hit:
declare -A files
wpath=/tmp
while :; do
while read -a crtfile; do
if [ "${crtfile:0:1}" = "-" ] && [ "${crtfile[8]:0:4}" = "test" ] &&
( [ "${crtfile[8]##*.}" = "css" ] || [ "${crtfile[8]##*.}" = "php" ] ) &&
[ "${files[${crtfile[8]}]:-0}" -lt ${crtfile[4]} ]; then
printf "\e[47m## %-14s :- %(%a %d %b %y %T)T ##\e[0m\n" ${crtfile[8]} -1
tail -c +$[1+${files[${crtfile[8]}]:-0}] $wpath/${crtfile[8]}
files[${crtfile[8]}]=${crtfile[4]}
fi
done < <(/bin/ls -l $wpath)
sleep 1
done
This will each seconds
for all entries in watched directory
search for files (first caracter is -),
search for filenames begining by test,
search for filenames ending by css or php,
compare already printed sizes with new file size,
if new size greater,
print out new bytes by using tail -c and
store new already printed size
sleep 1 seconds
this may output something like:
## test.css :- Thu 28 Apr 16 12:00:09 ##
Thu 28 Apr 16 12:00:02
BOFH excuse #216:
What office are you in? Oh, that one. Did you know that your building was built over the universities first nuclear research site? And wow, aren't you the lucky one, your office is right over where the core is buried!
Thu 28 Apr 16 12:00:05
BOFH excuse #145:
Flat tire on station wagon with tapes. ("Never underestimate the bandwidth of a station wagon full of tapes hurling down the highway" Andrew S. Tannenbaum)
Thu 28 Apr 16 12:00:06
BOFH excuse #301:
appears to be a Slow/Narrow SCSI-0 Interface problem
## test.php :- Thu 28 Apr 16 12:00:09 ##
Thu 28 Apr 16 12:00:03
BOFH excuse #36:
dynamic software linking table corrupted
Thu 28 Apr 16 12:00:08
BOFH excuse #367:
Webmasters kidnapped by evil cult.
## test.css :- Thu 28 Apr 16 12:00:10 ##
Thu 28 Apr 16 12:00:09
BOFH excuse #25:
Decreasing electron flux
## test.php :- Thu 28 Apr 16 12:00:13 ##
Thu 28 Apr 16 12:00:12
BOFH excuse #3:
electromagnetic radiation from satellite debris
Nota: If some file are modified more than one time between two checks, all modification will be printed on next check.
Although not really nice, the following gives (and repeats) the last 50 lines of the newest file in the current directory:
while true; do tail -n 50 $(ls -Art | tail -n 1); sleep 5; done
You can refresh every minute using a cronjob:
$crontabe -e
* * * * * /home/script.sh
if you need to refresh in less than a minute you can use the command "sleep" inside your script.
I am stuck.
I am wondering how to check if a Q-ID is a valid mailq ID.
/var/spool/mqueue (3 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
m9TMLQHG012749 1103 Thu Oct 30 11:21 <apache#localhost.localdomain>
(host map: lookup (electrictoolbox.com): deferred)
<test#electrictoolbox.com>
m9TMLRB9012751 37113 Thu Oct 30 11:21 <apache#localhost.localdomain>
(host map: lookup (electrictoolbox.com): deferred)
<test#electrictoolbox.com>
m9TMLPcg012747 240451 Thu Oct 30 11:21 <apache#localhost.localdomain>
(host map: lookup (electrictoolbox.com): deferred)
<test#electrictoolbox.com>
Total requests: 3
thanks for any hint.
Based on your last comment, I guess this is what would suffice.
randomMailQId=XXXXXXXX # get this populated
if mailq | grep "^$randomMailQId\s"; then
echo Valid
else
echo Invalid
fi
sample procmailrc:
SHELL=/bin/bash
LOGFILE=$HOME/procmail.log
VERBOSE=yes
:0
* ^Subject: envdump please$
{
LOG="`id`"
:0
/dev/null
}
/etc/group file contains (note the other usernames are vain attempts to make this work):
someuser:x:504:
s3:x:505:someuser,someotheruser,postfix,postdrop,mail,root
If I run as "someuser" the command id:
[someuser#lixyz-pqr ~]$ id
uid=504(someuser) gid=504(someuser) groups=504(someuser),505(s3)
However when I run procmail by sending an email with the subject "envdump please", the 505/s3 group disappears (this is in procmail.log):
procmail: [17618] Mon Dec 19 17:39:50 2011
procmail: Match on "^Subject: envdump please$"
procmail: Executing "id"
procmail: Assigning "LOG=uid=504(someuser) gid=504(someuser) groups=504(someuser)"
uid=504(someuser) gid=504(someuser) groups=504(someuser)procmail: Assigning "LASTFOLDER=/dev/null"
this server is running Fedora 14 with Postfix 2.7.5
Procmail wasn't installed setuid.
for background, it should look like:
[root#li321-238 postfix]# ls -l /usr/bin/procmail
-rwsr-sr-x. 1 root mail 92816 Jul 28 2009 /usr/bin/procmail
which you can set up via:
chmod ug+s /usr/bin/procmail