Every time, my application crash a core dump file is not generated. I remember that few days ago, on another server it was generated. I'm running the app using screen in bash like this:
#!/bin/bash
ulimit -c unlimited
while true; do ./server; done
As you can see I'm using ulimit -c unlimited which is important if I want to generate a core dump, but it still doesn't generate it, when I got an segmentation fault.
How can I make it work?
This link contains a good checklist why core dumps are not generated:
The core would have been larger than the current limit.
You don't have the necessary permissions to dump core (directory and file). Notice that core dumps are placed in the dumping process' current directory which could be different from the parent process.
Verify that the file system is writeable and have sufficient free space.
If a sub directory named core exist in the working directory no core will be dumped.
If a file named core already exist but has multiple hard links the kernel will not dump core.
Verify the permissions on the executable, if the executable has the suid or sgid bit enabled core dumps will by default be disabled. The same will be the case if you have execute permissions but no read permissions on the file.
Verify that the process has not changed working directory, core size limit, or dumpable flag.
Some kernel versions cannot dump processes with shared address space (AKA threads). Newer kernel versions can dump such processes but will append the pid to the file name.
The executable could be in a non-standard format not supporting core dumps. Each executable format must implement a core dump routine.
The segmentation fault could actually be a kernel Oops, check the system logs for any Oops messages.
The application called exit() instead of using the core dump handler.
Make sure your current directory (at the time of crash -- server may change directories) is writable. If the server calls setuid, the directory has to be writable by that user.
Also check /proc/sys/kernel/core_pattern. That may redirect core dumps to another directory, and that directory must be writable. More info here.
For systemd systems1, install the package systemd-coredump. Coredumps can be found via:
ls /var/lib/systemd/coredump
Furthermore, these coredumps are compressed in the lz4 format. To decompress, you can use the package liblz4-tool like this: lz4 -d FILE. To be able to debug the decompressed coredump using gdb, I also had to rename the utterly long filename into something shorter...
1 Debian 9 Stretch
Check:
$ sysctl kernel.core_pattern
to see how your dumps are created (%e will be the process name, and %t will be the system time).
For Ubuntu, dumps are created by apport in /var/crash, but in different format (see inside file).
You can test it by:
sleep 10 &
killall -SIGSEGV sleep
If core dumping is successful, you will see “(core dumped)” after the segmentation fault indication.
Read more:
How to generate core dump file in Ubuntu
https://wiki.ubuntu.com/Apport
Remember if you are starting the server from a service, it will start a different bash session so the ulimit won't be effective there. Try to put this in your script itself:
ulimit -c unlimited
If one is on a Linux distro (e.g. CentOS, Debian) then perhaps the most accessible way to find out about core files and related conditions is in the man page. Just run the following command from a terminal:
man 5 core
Also, check to make sure you have enough disk space on /var/core or wherever your core dumps get written. If the partition is almos full or at 100% disk usage then that would be the problem. My core dumps average a few gigs so you should be sure to have at least 5-10 gig available on the partition.
Note: If you have written any crash handler yourself, then the core might not get generated. So search for code with something on the line:
signal(SIGSEGV, <handler> );
so the SIGSEGV will be handled by handler and you will not get the core dump.
The answers given here cover pretty well most scenarios for which core dump is not created. However, in my instance, none of these applied. I'm posting this answer as an addition to the other answers.
If your core file is not being created for whatever reason, I recommend looking at the /var/log/messages. There might be a hint in there to why the core file is not created. In my case there was a line stating the root cause:
Executable '/path/to/executable' doesn't belong to any package
To work around this issue edit /etc/abrt/abrt-action-save-package-data.conf and change ProcessUnpackaged from 'no' to 'yes'.
ProcessUnpackaged = yes
This setting specifies whether to create core for binaries not installed with package manager.
If you call daemon() and then daemonize a process, by default the current working directory will change to /. So if your program is a daemon then you should be looking for a core in / directory and not in the directory of the binary.
Although this isn't going to be a problem for the person who asked the question, because they ran the program that was to produce the core file in a script with the ulimit command, I'd like to document that the ulimit command is specific to the shell in which you run it (like environment variables). I spent way too much time running ulimit and sysctl and stuff in one shell, and the command that I wanted to dump core in the other shell, and wondering why the core file was not produced.
I will be adding it to my bashrc. The sysctl works for all processes once it is issued, but the ulimit only works for the shell in which it is issued (maybe also the descendents too) - but not for other shells that happen to be running.
Just in case someone else stumbles on this. I was running someone else's code - make sure they are not handling the signal, so they can gracefully exit. I commented out the handling, and got the core dump.
In centos,if you are not root account to generate core file:
you must be set the account has a root privilege or login root account:
vim /etc/security/limits.conf
account soft core unlimited account hard core unlimited
then if you in login shell with securecrt or other:
logout and then relogin
Allow Dump from Daemons
To allow all daemons witch are started by systemd to core dump.
Edit: /etc/systemd/system.conf add following
DefaultLimitCORE=infinity
Edit: /etc/sysctl.d/core.conf add following
kernel.core_pattern = /var/lib/coredumps/core-%e-sig%s-user%u-group%g-pid%p-time%t
kernel.core_uses_pid = 1
fs.suid_dumpable = 2
more detail: https://pve.proxmox.com/wiki/Enable_Core_Dump_systemd
Our application stopped producing core dumps when a capability was set to it.
setcap 'cap_sys_nice=eip' /usr/bin/${our_app}
Removing it allowed the re-generation of coredumps.
setcap '-r' /usr/bin/${our_app}
See also: How do I get a coredump from a setcap executable?
Related
I am performing Forensic analysis on Host based evidence - examining partitions of a hard drive of a server.
I am interested in finding the processes all the "users" ran before the system died/rebooted.
As this isn't live analysis I can't use ps or top to see the running processes.
So, I was wondering if there is a log like /var/log/messages that shows me what processes users ran.
I have gone through a lot of logs in /var/log/* - they give me information about logins, package updates, authorization - but nothing about the processes.
If there was no "command accounting" enabled, there is no.
Chances to find something are not too big, anyway a few things to consider:
depends how gracefull death/reboot was (if processes were killed gracefully, .bash_history and similar files may be updated with recent session info)
utmp and wtmp files may give the list of active users at the reboot.
OS may be saving crash dump (depends on linux distribution). If so - You may be able to examine OS state at the moment of crash. See RedHat's crash for details (http://people.redhat.com/anderson/crash_whitepaper/).
/tmp, /var/tmp may hold some clues, what was running
any files with mtime and ctime timestamps (maybe atime also) near the time of crash
maybe You can get something usefull from swap partition (especially if reboot was related to heavy RAM usage).
So, I was wondering if there is a log like /var/log/messages that
shows me what processes users ran
Given the OS specified by the file system path of /var/log, I am assuming you are using ubuntu or some linux based server and if you are not doing live forensics while the box was running or memory forensics (where a memory capture was grabbed), AND you rebooted the system, there is no file within /var/log that will attribute processes to users. However, if the user was using the bash shell, then you could check the .bash_history file that shows the commands that were run by that user which I think is 500 (by default for the bash shell).
Alternatively, if a memory dump was made (/dev/mem or /dev/kmem), then you could used volatility to pull out processes that were run on the box. But still, I do not think you could attribute the processes to the users that ran them. You would need additional output from volatility for that link to be made.
I'm developing a project where the executables use Linux's POSIX capabilities rather than being setuid root. So far I've had to keep one root shell open so that each time I recompile I can redo the setcap command to give the needed capability to the executable file so that I can test the results. That's getting tedious, plus if I ever hope that anyone else would want to contribute to the project's development I'll have to come up with a better way of doing it.
So far I've come up with two ways of dealing with this:
1) Have a single make target to be run as root to create a special setuid program which will be used to by the makefiles to give the capability to the executables. The program will be compiled from a template modified via sed so that it will only run if used by the non-root user the developer is working as, and will only modify files owned by the developer (and which are sitting in directories owned by the developer which aren't world writeable).
The problem with this is that I'm using GNU autotools to generate my make files, and I can't figure out how to get the makefiles to run a program on a linked executable after it's been linked. I could create a setcap-all target which has all the executables as its dependencies, with a rule that runs the setuid program on them, but then you can't simply do make executable-1 if that's all you want to build.
2) Have a single make target to be run as root to create a setuid daemon which will use inotify to monitor the src directory and grant the capability to any new executables (and which has security consideration similar to the setuid program from #1).
My problem with this is that I can't figure out how to get the build system to automatically and transparently start up the daemon, plus my intuition that This Is Not The Way Things Are Done in a proper build system.
Are there any better ways of doing this?
Maybe I'm a bit confused about the question, but it seems you're trying to use the build-system to solve an installation problem.
Whether you're packaging your project using dpkg, rpm or anything else, there should be a rule to enforce usage of setcap, which will set the capabilities of the installed binary using the Filesystem Extended Attributes (xattrs).
# Post-install rule example
setcap cap_net_raw=+pe /usr/bin/installed-binary
However, of you're installing a system daemon, you may count on the init-script to already have all the capabilities, so it's a matter of letting your process to drop unneeded capabilities.
I've got a few processes that talk to each other through named pipes. Currently, I'm creating all my pipes locally, and keeping the applications in the same working directory. At some point, it's assumed that these programs can (and will) be run from different directories. I need to create these pipes I'm using in a known location, so all of the different applications will be able to find the pipes they need.
I'm new to working on Linux and am not familiar with the filesystem structure. In Windows, I'd use something like the AppData folder to keep these pipes. I'm not sure what the equivalent is in Linux.
The /tmp directory looks like it probably could function just nicely. I've read in a few places that it's cleared on system shutdowns (and that's fine, I have no probably re-creating the pipes when I start back up.) but I've seen a few other people say they're losing files while the system is up, as if it's cleaned periodically, which I don't want to happen while my applications are using those pipes!
Is there a place more suited for application specific stores? Or would /tmp be the place that I'd want to keep these (since they are after all, temporary.)?
I've seen SaltStack using /var/run. The only problem is that you need root access to write into that directory, but let's say that you are going to run your process as a system daemon. SaltStack creates /var/run/salt at the installation time and changes the owner to salt so that later on it can be used without root privileges.
I also checked the Filesystem Hierarchy Standard and even though it's not really important so much, even they say:
System programs that maintain transient UNIX-domain sockets must place them in this directory.
Since named pipes are something very similar, I would go the same way.
On newer Linux distros with systemd /run/user/<userid> (created by pam_systemd during login if it doesn't already exist) can be used for opening up sockets and putting .pid files there instead of /var/run where only root has access. Also note that /var/run is a symlink to /run so /var/run/user/<userid> can also be used. For more infos check out this thread. The idea is that system daemons should have a /var/run/<daemon name>/ directory created during installation with proper permissions and put their sockets/pid files in there while daemons run by the user (such as pulseaudio) should use /run/user/<userid>/. Another option is /tmp and /var/tmp.
I have a linux service (c++, with lots of loadable modules, basically .so files picked up at runtime) which from time to time crashes ... I would like to get behind this crash and investigate it, however at the moment I have no clue how to proceed. So, I'd like to ask you the following:
If a linux service crashes where is the "core" file created? I have set ulimit -c 102400, this should be enough, however I cannot find the core files anywhere :(.
Are there any linux logs that track services? The services' own log obviously is not telling me that I'm going to crash right now...
Might be that one of the modules is crashing ... however I cannot tell which one. I cannot even tell which modules are loaded. Do you know how to show in linux which modules a service is using?
Any other hints you might have in debugging a linux service?
Thanks
f-
Under Linux, processes which switch user ID, get their core files disabled for security reasons. This is because they often do things like reading privileged files (think /etc/shadow) and a core file could contain sensitive information.
To enable core dumping on processes which have switched user ID, you can use prctl with PR_SET_DUMPABLE.
Core files are normally dumped in the current working directory - if that is not writable by the current user, then it will fail. Ensure that the process's current working directory is writable.
0) Get a staging environment which mimics production as close as possible. Reproduce problem there.
1) You can attach to a running process using gdb -a (need a debug build of course)
2) Make sure the ulimit is what you think it is (output ulimit to a file from the shell script
which runs your service right before starting it). Usually you need to set ulimit in /etc/profile file; set it ulimit -c 0 for unlimited
3) Find the core file using find / -name \*core\* -print or similar
4) I think gdb will give you the list of loaded shared objects (.so) when you attach to the process.
5) Add more logging to your service
Good luck!
Your first order of business should be getting a core file. See if this answer applies.
Second, you should run your server under Valgrind, and fix any errors it finds.
Reproducing the crash when running under GDB (as MK suggested) is possible, but somewhat unlilkely: bugs tend to hide when you are looking for them, and the debugger may affect timing (especially if your server is multi-threaded).
Is it possible to start a process in Linux, and restrict its access to certain files/directories? For example:
$ start-process --enable-dir=./sandbox --exec="some-script.sh"
some-script.sh won't be able to do anything outside of ./sandbox.
You can use chroot to set the root directory of your process tree. This means however, that all dependencies of that process must be available in it's new root.
There are a number of packages that can help you setup chroot-environments for your needs. Google is your friend ;)
Some pointers on building a chroot environment
When builing a chroot for some program or daemon you have to have a complete environment for the program you want to chroot. This means you have to provide a minimum system in a directory. That might contain:
A shell and some shell utilities, or a variant of busybox. (this encompasses the next step too, if you aren't planning on deploying one single static executable that is).
Libc and other dependent shared libraries.
You need to check shared library dependencies using ldd or objdump. Every library that appears has to be in your private root directory. This step might be repeated several times for every executable and library you need. Note that some libraries, which are linked explicitly at runtime using dlopen need to be checked separately.
Depending on what you plan to chroot a minimal /dev tree.
If you plan to chroot a daemon process this may well be needing some minimal files in /dev such as random or zero. You can create those with the mknod command. Please refer to the mknod documentation, as well as the linux documentation on which major/minor numbers which device should have.
Also depending on what you plan to chroot is a minimal /etc. Files needed therein are:
A minimal passwd and shadow (not your system passwd/shadow).
A minimal mtab containing /.
A minimal group (again, not your system group file).
You have to start somewhere, so it's best to start with the prerequisites for you program. Refer to your documentation for specifics.
Typically you want to chroot the process, so that it can only access a directory and its sub-directories, and only execute some defined commands.
See How to chroot.