Nagios Custom Script for finding defunct processes - linux

Question

Objective: Check for zombie processes, but if any are found to be owned by the account nagios, ignore them. If a zombie process is found to be owned by an account other than the nagios account, then issue an alert.
Currently have the following nagios check in the nrds.cfg to find zombie processes.
command[Check_ZombieProcs]=/opt/tools/nagitem/libexec/check_procs -w 1 -c 2 -s Z
Which will generate output like this when it finds something:
PROCS CRITICAL: 3 processes with STATE = Z | procs=3;1;2;0;
To find out who owns the above following 3 defunct zombie processes, I can manually run the command:
ps -elf | grep Z | grep "defunct" which results in the output shown below.
0 Z nagios 9487 9948 0 80 0 - 0 do_exi Nov07 ? 00:00:00 [sh] <defunct>
0 Z nagios 28647 9949 0 80 0 - 0 do_exi Nov03 ? 00:00:00 [sh] <defunct>
0 Z nagios 67429 9947 0 80 0 - 0 do_exi Nov03 ? 00:00:00 [sh] <defunct>
My problem is that the defunct processes shown above end up clearing on their own and I want to exclude from the output any defunct zombie processes owned by the account nagios.
Since the check_procs nagios plugin does not have an argument to handle the exclusion of a user, I am creating the following script.
With the script below, what I want to accomplish is:
If status is equal to null or is equal to nagios, then state should be ok (green).
If status is equal to anything other than null or nagios, then state should be critcal (red).
#!/bin/ksh
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3
SCRIPTPATH=`echo $0 | /bin/sed -e 's,[\\/][^\\/][^\\/]*$,,'`
if [[ -f ${SCRIPTPATH}/utils.sh ]]; then
. ${SCRIPTPATH}/utils.sh # use nagios utils to set real STATE_* return values
fi
printvariables() {
echo "Variables:"
#Add all your variables at the end of the "for" line to display them in verbose
for i in EXITSTATUS EXITMESSAGE
do
echo -n "$i : "
eval echo \$${i}
done
echo
}
#Set to unknown in case of unexpected exit
EXITSTATUS=$STATE_UNKNOWN
EXITMESSAGE="UNKNOWN: Unexpected exit. You should check that everything is alright"
ZOMB=$(ps -elf | grep Z | grep "defunct" | awk {'print $3'} | sort -u)
if [ "$ZOMB" == "nagios" ];then
EXITSTATUS=$STATE_OK
EXITMESSAGE="OK - No Defunct processes found"
elif [ "$ZOMB" == "" ];then
EXITSTATUS=$STATE_OK
EXITMESSAGE="OK - No Defunct processes found"
else
EXITSTATUS=$STATE_CRITICAL
EXITMESSAGE="CRITICAL - Defunct processes found owned by $ZOMB"
fi
#Script end, display verbose information
if [[ $VERBOSE -eq 1 ]] ; then
printvariables
fi
echo ${EXITMESSAGE}
exit $EXITSTATUS
The command:
ps -elf | grep Z | grep "defunct" | awk {'print $3'}
has the potential to give output similar to the following which no doubt will cause an issue. I cannot replicate or thoroughly test as I only occasionally see a defunct process.
nagios
root
someuser
Note that in my script above I am using (see the sort -u):
ps -elf | grep Z | grep "defunct" | awk {'print $3'} | sort -u
to eliminate getting the same account (like nagios) listed multiple times in the output. My script above does work, but I am concerned that it cannot handle if there is more than one account that has defunct processes.
Appreciate any help.

Answer 1

This is more of a bash question than a nagios question, but I'll bite.
The first issue I see is that you're simply pushing all output from ps into $ZOMB when what you probably want to do is to iterate over each line (process).
You could do something like this:
For each line in the ps output, check if it's a zombie process. If not, discard it.
If yes, check whether the user is nagios, if so discard it.
If not, we have a problem, so exit 1 and say what the user and process is.
Example:
#!/bin/bash
echo "$(ps -elf)" | while IFS= read -r line ;
do
user=$(echo "$line" | awk {'print $3'})
stat=$(echo "$line" | awk {'print $2'})
proc=$(echo "$line" | awk {'print $15'})
if [ "$stat" != "R" ]; then
# it's not running, we don't care
continue
elif [ "$user" == "root" ]; then
# it's root, we don't care
continue
else
# we do care! print info and exit 1 immediately
echo "Found process: $user '$proc'"
exit 1
fi
done
In your case, "R" would be "Z" and "root" would be "nagios".