Is there a place where the Linux kernel passively logs SIGKILL (kill -9) shutdown requests?
I have a JVM running that is arbitrarily being shut down and I suspect that, based on the evidence available, is being shut down by a stray process that is somehow issuing a shutdown of the JVM process. I have robust logging in place but in order to confirm my suspicion, I'd have to turn up the logging level to overwhelming levels.
I've researched exhaustively through /var/log and can't seem to find any place that might capture and log these SIGKILL events. Any ideas where I might find these events, if they exist?
Option 1:
If your kernel has ftrace support (very likely) try the killsnoop tool from Brendan Gregg's perf-tools:
wget https://raw.githubusercontent.com/brendangregg/perf-tools/master/killsnoop
chmod +x killsnoop
sudo ./killsnoop -s
More usage examples in the killsnoop_example.txt file.
Option 2: (passive capture)
If your kernel has no ftrace support you can use the kernel-siglog kernel module from https://github.com/nfedera/kernel-siglog :
git clone https://github.com/nfedera/kernel-siglog.git
cd kernel-siglog/
make
sudo insmod siglog.ko
Once inserted the siglog kernel module will record the last 10,000 signals in /proc/siglog
I had a similar issue and found the culprit using this kernel module. I had it inserted on a customer's server for some weeks and when the service was killed I logged in, did a cat /proc/siglog and found that my service was killed by a customer's own buggy watchdog script.
Related
I have a VPS with firewall and security notices enabled. I keep getting emails like this:
Time: Wed Jun 19 19:01:54 2019 -0500
Account: user
Resource: Process Time
Exceeded: 7248 > 3600 (seconds)
Executable: /opt/cpanel/ea-php72/root/usr/sbin/php-fpm
Command Line: php-fpm: pool domain_com
PID: 16374 (Parent PID:9915)
Killed: No
So for some reason with this example I have a script that has apparently been running for 2+ hours non-stop. I don't have anything that should be doing that.
I'm getting notices like this quite often. How can I use this info to track down what specifically is causing this?
Any information would be greatly appreciated. Thanks!
You can track which the exact process with the process ID mentioned.
lsof -p 16374
The alert which you are getting is from the LDF which is installed as a part of CSF. I think its normal for cPanel with php_fpm to have the process php_fpm run this long.
You can add the php-fpm to csf.pignore file to stop this warning.
You can also refer the below cPanel fourm thread.
https://forums.cpanel.net/threads/lfd-excessive-resource-usage-normal-for-php-fpm.592583/
To get more information on processes, I would use the Htop tool. This is a great article for learning about how to manage processes using htop and ps
Lsof (List open files) will tell you more information about what files the process is using.
You can get htop and lsof with
sudo apt install htop lsof -y
This article indicates that :
That message comes from the third-party CSF/LFD application and indicates a PHP-FPM process was running longer than the maximum time configured for the CSF/LFD detection period. It shows the process was not killed, thus you should not have traffic loss.
So you might want to check the PHP-FPM error log for the account in-question to see if you notice any particular error messages. It's located at:
/home/$username/logs/domain_tld.php.error.log
It looks like your specific issue has not been resolved on that form. So, you might want to try strace. It handles watching system calls made by a given process including all read-write operations and os function calls. You can activate it on the command line before the program you want to track or attach to a running process by hitting s on a process selected in htop.
I newbie to linux/fedora and i'm doing some network scanning tests (i took a pen-testing course in Udemy ) so for better scanning i killed the following processes, in that specific order:
NetworkManager
dhclient
wpa_supplicant
avahi-daemon
I've check with
airodump-ng wlp2s0 check //for verifying that the processes are killed
and they do - now i wish to Enable them all back,
so i run the process name but couldn't get it to work - the only process that i was manage to enable was NetworkManager - all the rest return errors which i guess - suggest i doing it wrong.... perphas the order i did so was wrong...
We deployed multiple JVMs in a Linux System along with a Node.js and a mongod server instance. It is https enabled micro services architecture.
Some of the above listed processes are getting down randomly. There is no clue why they are down.
No OOM related log trace in /var/log syslog files, Is it possible for linux to kill the process other than OOM.
What are the possible syslog errors ? ( I have the complete /var/log files )
Where to look to identify the actual issue ? We have no clue on this... Sombody can help please to trace the process end to end
Assuming you are starting these JVM processes in the background, did you use nohup to make sure they ignore SIGHUP? Also it would be wise to redirect the output of these processes to the output files.
E.g.
nohup node server1.js >& server1.out &
And then you can investigate in those files for any possible messages if there are any.
I'm running linux on a mips based system (specifically openwrt on a router).
When I run the reboot (as supplied by busybox) i.e. just reboot on it's own, the system reboots, but some of the services (webserver, dhcp/dns, dsl stuff) don't start up.
However when I reboot via the web interface, all the services start normally. I looked at the code and saw that the web interface runs reboot > /dev/null 2>&1. Running this command also reboots and starts up0 the services properly.
My question is how does redirecting stdout and stderr to /dev/null affect the startup of services upon the next boot?
Also, I'm wondering, would reboot contain architecture specific code?
No, redirecting stdout/stderr must not be able to affect the boot process (and where would that be saved anyway?). There must be something else causing this.
Does "shutdown -r now" work?
Since I install pending updates for my Ubuntu server as soon as possible, I have to restart my linux server quite often. I'm running an webapp on that server and would like to warn my users about the pending restart. Right now, I do this manually, adding an announcement before the restart, give them some time to finish their work, restart and remove the announcement.
I hope, shutdown -r +60 writes an file with all the information about the restart, which I can check on every access. Is there such a file? Would prefer a file in a virtual file system like /proc for performance reasons...
I'm running Ubuntu 10.04.2 LTS
If you are using systemd, the following command shows the scheduled shutdown info.
cat /run/systemd/shutdown/scheduled
Example of output:
USEC=1636410600000000
WARN_WALL=1
MODE=reboot
As remarked in a comment by #Björn, USEC is the timestamp in micro seconds.
You can convert it to a human friendly format dropping the last 6 figures and using date like this:
$ date -d #1636410600
Mon Nov 8 23:30:00 CET 2021
The easiest solution I can envisage means writing a script to wrap the shutdown command, and in that script create a file that your web application can check for.
As far as I know, shutdown doesn't write a file to the underlying files system, although it does trigger broadcast messages warning of the shutdown, which I suppose you could write a program to intercept .. but the above solution seems the easiest.
Script example:
shutdown.bsh
touch /somefolder/somefile
shutdown -r $1
then check for 'somefile' in your web app.
You'd need to add a startup link that erased the 'somefile' otherwise it would still be there when the system comes up and the web app would always be telling your users it was about to shut down.
You can simply check for running shutdown process:
if ps -C shutdown > /dev/null; then
echo "Shutdown is pending"
else
echo "Shutdown is not scheduled"
fi
For newer linux distributions versions you might need to do:
busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager ScheduledShutdown
The method of how shutdown works has changed
Tried on:
- Debian Stretch 9.6
- Ubuntu 18.04.1 LTS
References
Check if shutdown schedule is active and when it is
The shutdown program on a modern systemd-based Linux system
You could write a daemon that does the announcement when it catches the SIGINT / SIGQUIT signal.