Using lighttpd (version 1.4.35) on Linux: How can I get a core dump from a CGI program that exits with SIGABORT (or SIGSEGV for that matter) instead of lighttpd trapping the signal?
Are there lighttpd configuration options to prevent lighttpd or mod_cgi from trapping the signals?
I can modify & recompile the CGI program to reset the signal handler of interest, but I would rather do this with modifying the code in question.
EDIT: Here's the message I see from lighttpd:
(mod_cgi.c.1312) cleaning up CGI: process died with signal 6
mod_cgi.c.1050: aborted
Related
How to have Linux wait till my C++ program completes its cleanup routine. The program initially calls function sigaction(2) to register a custom SIGTERM handler. If test the handler by running kill -s TERM $(ps -C a.out -o pid=), it would have no problem. However, in the real shutting down is another case. Sometimes the handler can get its job all done, but sometimes not. Apparently there is a race condition when the machine is shutting down. Does anyone know how to make the system wait a little bit longer so as to avoid the race condition? Thanks.
In the comments you said you use MX Linux version 18.2. It seems to be based on Debian 9, which uses systemd by default, but still has the option to revert to classic SysVinit if desired. The web pages of MX Linux seem to emphasize the UI and do not mention anything special about the init system, so I'll assume it uses systemd too.
With systemd, a mechanism called control groups (cgroups for short) is in play: when systemd starts a service, it will also place its process in a special cgroup. This cgroup is automatically inherited by any process started by the service. When systemd is stopping a service, it will first execute any custom ExecStop actions, if any are defined for the service, wait for TimeoutStopSec, and if there are then any processes left in the service's cgroup, systemd will send them a SIGTERM, wait for another TimeoutStopSec, and then will send a SIGKILL for any left-over processes in the cgroup.
The thing tripping you up is probably the fact that user sessions are also encapsulated in a cgroup: anything you start manually, with e.g. sh /etc/init.d/yourservice start will still count as part of your user session, even if it performs all the actions needed to classically daemonize. And so, when you initiate a shutdown, the first action is to log out any user sessions... which causes your manually-started service to first receive a SIGHUP, then SIGTERM after a short delay, and possibly a SIGKILL after another short delay. Once the user sessions are cleaned up, the rest of the shutdown process will happen.
In order to use init.d scripts successfully with systemd, you'll need to know a few things.
systemd's compatibility mechanism for init.d scripts works by automatically generating a native systemd .service unit file for every init.d script, and then using those unit files just like native systemd services. This causes three requirements you might be unfamiliar with:
your init.d script should have a Linux Standard Base comment block before any non-comment lines in the script that describes the dependencies to any other services. It should look somewhat like this (example from Dovecot IMAP server):
### BEGIN INIT INFO
# Provides: dovecot
# Required-Start: $local_fs $remote_fs $network $syslog $time
# Required-Stop: $local_fs $remote_fs $network $syslog
# Should-Start: postgresql mysql slapd winbind nslcd
# Should-Stop: postgresql mysql slapd winbind nslcd
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Dovecot init script
# Description: Init script for dovecot services
### END INIT INFO
After placing your init.d script, you should run systemctl daemon-reload to trigger the re-run of the systemd-sysv-generator that produces the unit file that will call your script.
After placing the script as /etc/init.d/yourservice and running systemctl daemon-reload, you should start the service using either systemctl start yourservice or service yourservice start. Only these methods cause systemd to place your service to its own control group, which will be important for the orderly shutdown your service needs. Running sh /etc/init.d/yourservice start will not do that.
You can use systemctl cat yourservice to view the resulting auto-generated yourservice.service unit file.
It might be a better idea to just write a native systemd service unit file for your service. You can find the distribution's standard unit files at /lib/systemd/system/ directory; you can use them as examples, but you should place your custom unit file into /etc/systemd/system instead, so your unit file won't be overridden by any package updates.
We deployed multiple JVMs in a Linux System along with a Node.js and a mongod server instance. It is https enabled micro services architecture.
Some of the above listed processes are getting down randomly. There is no clue why they are down.
No OOM related log trace in /var/log syslog files, Is it possible for linux to kill the process other than OOM.
What are the possible syslog errors ? ( I have the complete /var/log files )
Where to look to identify the actual issue ? We have no clue on this... Sombody can help please to trace the process end to end
Assuming you are starting these JVM processes in the background, did you use nohup to make sure they ignore SIGHUP? Also it would be wise to redirect the output of these processes to the output files.
E.g.
nohup node server1.js >& server1.out &
And then you can investigate in those files for any possible messages if there are any.
Is there a place where the Linux kernel passively logs SIGKILL (kill -9) shutdown requests?
I have a JVM running that is arbitrarily being shut down and I suspect that, based on the evidence available, is being shut down by a stray process that is somehow issuing a shutdown of the JVM process. I have robust logging in place but in order to confirm my suspicion, I'd have to turn up the logging level to overwhelming levels.
I've researched exhaustively through /var/log and can't seem to find any place that might capture and log these SIGKILL events. Any ideas where I might find these events, if they exist?
Option 1:
If your kernel has ftrace support (very likely) try the killsnoop tool from Brendan Gregg's perf-tools:
wget https://raw.githubusercontent.com/brendangregg/perf-tools/master/killsnoop
chmod +x killsnoop
sudo ./killsnoop -s
More usage examples in the killsnoop_example.txt file.
Option 2: (passive capture)
If your kernel has no ftrace support you can use the kernel-siglog kernel module from https://github.com/nfedera/kernel-siglog :
git clone https://github.com/nfedera/kernel-siglog.git
cd kernel-siglog/
make
sudo insmod siglog.ko
Once inserted the siglog kernel module will record the last 10,000 signals in /proc/siglog
I had a similar issue and found the culprit using this kernel module. I had it inserted on a customer's server for some weeks and when the service was killed I logged in, did a cat /proc/siglog and found that my service was killed by a customer's own buggy watchdog script.
On my raspberry pi, I want my own written server to be started at startup, and to be restarted when it segfaults, so I added it to /etc/inittab. The problem is that the server won't start
The line I added:
1:2345:respawn:/home/gear/lionfish/main /home/gear/lionfish/app
When I run this command from the command line it works just fine, but the server doesn't run. I've checked this with ps aux, and it didn't show up
Have I made some sort of mistake?
EDIT: Small side question. The server needs root privileges, does inittab do this automatically or do I need to add something to it?
Typical problems:
As already mentioned, environment is set up differently. Make sure $PATH iscorrect.
Does your program try to execute in a directory which becomes unmounted? If so, cd to / first.
Access restrictions to files and directories.
Process doesn't detach from stdin/stdout/stderr.
The process runs in foreground instead of background.
Parent process receives a terminating signal such as SIGTERM which kills your process as well. Try ignoring this (and some others) signals by using nohup or sigset/sigignore.
Debugging hint: Let the server start by appending current time to the end of an already existing file in a directory which is guaranteed to be writable. Make sure you flush (and close) the file pointer immediately. Then at least you can see whether it was started at all or not.
I try to use "valgrind --vgdb=yes httpd -X" to debug a service running via httpd, and monitor memory leaks using "monitor leak_check full reachable increased" after each request. The problem is that I cannot interrupt httpd reliably in order to run the "monitor" command in gdb - when I press Ctrl-C in valgrind/httpd terminal, it usually terminates, same when I send kill -INT. I tried different "handle SIGINT/SIGTERM nopass/pass" but with no luck.
Also often, when I just add breakpoints and pause httpd in Eclipse CDT/gdb debugger, httpd terminates, too.
So is it possible to interrupt httpd without making it exit in 90% cases?