I am using a customized OS based on Oracle Linux 6.10. I have few customized command that helps to find information like firewall rules,hardware information but if I run those command then coredumps are generated.
There are 4 core files found on the one of my OracleLinux6.10 server like core.top.myrole.34089.bt
Below are the error that say it is generated by top.myrole 'Program terminated with signal 6, Aborted/No symbol table info available':
10/15/2019 13:54:57 core.top.myrole.34089.bt
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
top.myrole: No such file or directory.
Missing separate debuginfo for the main executable file
Try: yum --enablerepo='*-debug*' install
/usr/lib/debug/.build-id/81/7805fb3de3731d5c36c8f7da38040ee6d510ea
[New Thread 34090]
[New Thread 34089]
Core was generated by \top.myrole'.`
Program terminated with signal 6, Aborted.
#0 0x000000347960f00d in ?? ()
Thread 2 (Thread 34089):
#0 0x000000347960f00d in ?? ()
No symbol table info available.
#1 0x0000000000000001 in ?? ()
No symbol table info available.
After the above line 'No symbol table' error would now repeat for multiple threads.
Then found other coredump also generated while running a customized command 'hardwareshow'(this command is not linux command but it is customized within our distribution). The log shows that core was generated by '/usr/bin/perl /usr/product/bin/iprule show' :
10/14/2019 20:44:25, core.hardwareshow.8944.bt
warning: core file may not match specified executable file.
[New Thread 55835]
[Thread debugging using libthread_db enabled]
Core was generated by \/usr/bin/perl /usr/product/bin/iprule show--type=rule --protocol=ipv6 -'.Program terminated with signal 11, Segmentation fault.#0 0x0000003479a80829 in ?? () from /usr/lib64/perl5/CORE/libperl.so`
------------
10/14/2019 04:19:59, core.iprule.55835.bt
I checked the Perl version used in the machine and it seems it is Perl5 , (still needs to confirm if it`s 5.10).How do I address this issue, I heard from some Perl experts that Perl upgrade to a new version can help here still I would want to hear from multiple Linux developers/Perl Developers about this error.
The perl interpreter will coredump if: there is a bug in its implementation; if certain pathological Perl code patterns are used (e.g. infinite recursion calling tie or overload handlers); or if there is a bug in an external module that uses C-level code (like XS modules). Determining what the fault is can be hard. If you just want the problem to go away, then upgrading everything to a more recent version is probably the path of least resistance. Note that the perl 5.10.0 release is about 12 years old.
Related
I have setup Linux Kernel debug environment with VMware Workstation. But When I tried to connect with gdb that connects correctly but I can't set any breakpoint or examine any kernel symbol.
Target Machine (debugee) Ubuntu 18:
I have compiled linux kernel 5.0-0 with the following directives:
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_FS=y
# CONFIG_DEBUG_SECTION_MISMATCH is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
Also my VMX file configuration:
debugStub.listen.guest64 = "TRUE"
debugStub.listen.guest64.remote="TRUE"
After that I transfered vmlinux to debugger machine and use gdb:
bash$ gdb vmlinux
gdb-peda$ target remote 10.251.31.28:8864
Remote debugging using 10.251.31.28:8864
Warning: not running or target is remote
0xffffffff9c623f36 in ?? ()
gdb-peda$ disas sys_open
No symbol "do_sys_open" in current context.
First you need to install kernel-debug-devel, kernel-debuginfo, kernel-debuginfo-common for corresponding kernel version.
Then you can use crash utility to debug kernel, which internally uses gdb
The symbol name you're looking for is sometimes not exactly what you expect it to be. You can use readelf or other similar tools to find the full name of the symbol in the kernel image. These names sometimes differ from the names in the code because of various architecture level differences and their related header and C definitions in kernel code. For example you might be able to disassemble the open() system call by using:
disas __x64_do_sys_open
if you've compiled it for x86-64 architecture.
Also keep in mind that these naming conventions are subject to change in different versions of kernel.
I have a arm executable[ (Debug build) ELF 32 bit LSB executable, ARM version (SYSV)] process executing on ARM Cortex A9 target having Linux OS(KErnel 2.6.38.8 )
The process has user id root groupid root
Even when the process crashes after getting SIGSEGV, there is no core file generated.
Now I have read this question to ensure it has file system is writable, ulimit -c is unlimited, user is root and has permissions, but still something is missing.
Here are few outputs of certain varaibles of my process and system, related to a core file creation :
cat /proc//coredump_filter is 00000033
cat /proc/sys/kernel/core_pattern is core
cat /proc/sys/kernel/core_uses_pid is 0
I have tried everything but stumped.
Could there be any kernel config/build option disabling the core creation?
Any other pointers?
EDIT:
I did simple test as below and it created a core file, but my process crash still does not dump core file.
sleep 20 &
killall -SIGSEGV sleep
Could there be any kernel config/build option disabling the core creation?
It is hidden under General Setup|Embedded System or General Setup|Configure standard... depending on your Linux version. The symbol value is ELF_CORE and it is in init/Kconfig. If it is not enabled, you will never get core dumps.
As suggested in a hidden comment in why coredump file is not generated.
I cross-compiled a Helloworld executable for ARM. It worked well on my friend's development board, but failed with a " segmentation fault " on my board. The two boards are slightly different in both hardware and software.
My question is, how can I debug in my board? Is it relatively easy to debug such a simple program? Does it indicate that there's nothing wrong with the executable and the problem most probably lies in the filesystem of my board?
Below is the code:
#include <QApplication>
#include <QPushButton>
int main(int argc, char *argv[])
{
QApplication app(argc, argv);
QPushButton hello("Hello world");
hello.resize(100, 30);
hello.show();
return app.exec();
}
And the executable is generated by the following commands:
qmake -project
qmake
make
most probably gdb is ported to be run on ARM target but in case lack of that or for easy debugging, you should use gdb remote debugging.
http://sourceware.org/gdb/onlinedocs/gdb/Remote-Debugging.html#Remote-Debugging
Gdbserver is the application should be run on target. the following is demonstration howto use it. (copied from wikipedia)
Target settings:
remote$ gdbserver :2345 hello_world
Process hello_world created; pid = 2509
Listening on port 2345
Host settings:
local$ gdb -q hello_world
Reading symbols from /home/user/hello_world...done.
(gdb) target remote 192.168.0.11:2345
Remote debugging using 192.168.0.11:2345
0x002f3850 in ?? () from /lib/ld-linux.so.2
(gdb) continue
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x08048414 in main () at hello_world.c:10
10 printf("x[%d] = %g\n", i, x[i]);
(gdb)
So you mentioned after LD_LIBRARY_PATH your issue was resolved. And before setting the LD_LIBRARY_PATH if your application gives error can not find libQt then it means you do not have Qt, but your application is giving this Seg Fault that mean you have library but not the right one so we can say you have multiple installations of Qt on your filesystem.
The one to which you have pointed now is correctly compiled for your current hardware but the other one is not compiled for your hardware causing the Segfault and this installation is in your library search path.
One possible reason of this seg fault can be determined from below.
Following are some CFLAGS which if not set correctly for any particular hardware, the compiled application / library causes Seg faults at run time.
-march
-mtune
-mfpu
So if your binary / library is compiled with say -march=armv5a and you are running it on ARM9 then it will crash like this.
Also note that not all application uses these flags, usually these flags are optimization flags and are used by base system libraries (i.e Qt, Glib, bison, Gtk etc....).
Even if you write a simple C based hello world application and your glibc is not compiled for your hardware you will get the Seg fault.
Answer from Author:
What caused this "segmentation fault" is exactly the software difference of the board. Specifically, the environmenat variable LD_LIBRARY_PATH was predefined in the failed board. And I added my path by the command
export LD_LIBRARY_PATH=$LD_LIBRARAY_PATH:/my/qt/path
Thus the predefined paths caused the problem ( still don't know in what way ).
If I change the command to
export LD_LIBRARY_PATH=/my/qt/path
the executable works.
As a general rule you shouldn't create objects derived from QObject on the stack as the QMetaObject system manages the life-time of objects with a parent-child relationship, and will therefore risk free()ing memory on the stack.
I get the following message in gdb (version 7.1):
[Thread debugging using libthread_db enabled]
and the command reverse-step
results with the following error message:
(gdb) reverse-step
Target multi-thread does not support this command
I am debugging a serial code right now, so I definitely do not need multi-threading. Can I turn this off somehow so that I get the latest reverse-debug commands to work? Also, if the code is parallelised with OpenMPI, there will be no need for multi-thread debugging at all, right?
Edit: Is this set as a compilation flag that can be just excluded?
You don't mention which version of GDB you're using, but since a little while, the parameter libthread-db-search-path is available.
(gdb) set libthread-db-search-path /tmp
(gdb) start
Temporary breakpoint 1 at 0x400632: file threads.c, line 14.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
will tell GDB to lookup it's helper library (libthread-db.so) in a directory where it isn't, so multithread debugging won't be enabled!
I'm not sure about OpenMPI parallel applications are multiprocesses (in contrast with OpenMP where they are multithreaded), so it won't change anything for you.
EDIT: Multithread debugging is usually only enabled when libpthread.so or equivalent is loaded by your process (ldd your-process to check if it's linked as a shared library) so if you don't need it, there might be a problem in your compilation script.
After a Xen guest domain hang, I took a dump using xm core-dump . Following the sparse documentation I found, I tried using the crash utility to analyze the dump.
Unfortunately, the kernel image (Debian lenny) is stripped, so I am forced to make use of the map file.
However,
crash
/boot/System.map-2.6.26-2-xen-amd64
vmlinux-2.6.26-2-xen-amd64
/mnt/my-core-file
(with vmlinux-2.6.26-2-xen-amd64 being the gunzip'ed vmlinuz image) fails:
crash: vmlinux-2.6.26-2-xen-amd64: no
debugging data available
Then I read that current Xen versions produce ELF-compatible dumps for guest domains. Indeed, this seems to be the case:
~$ sudo file my-core-dump
my-core-dump: ELF 64-bit LSB core file x86-64, version 1
However, gdb vmlinux-2.6.26-2-xen-amd64 my-core-dump fails, too:
...is not a core dump: File format not
recognized
Any hints?
Have you tried attaching to the domU console ?
xm create domU.conf -c
On the subject of the core-dump file, I found this:
http://lists.xensource.com/archives/html/xen-devel/2006-12/msg00456.html
I just want to check that you aren't under the impression that 'xm
dump-core' emits an Elf core file. It doesn't -- the format is custom and as
far as I know is only interpreted by a set of gdbserver patches that we ship
in our repository. Does the crash utility really support this special
format?
Edit: This might help to debug the core-dump: http://os-drive.com/files/docbook/xen-faq.html#setup_gdb