When I run perf test, one test that fails inside a Ubuntu 21.04 Docker container but succeeds on its Ubuntu 21.04 host is 26: Object code reading. It seems like it could be related to some issues with perf in the container, e.g. dwarf unwinding doesn't work at all.
The verbose output is
26: Object code reading :
--- start ---
test child forked, pid 92383
maps__set_modules_path_dir: cannot open /lib/modules/5.11.0-22-generic dir
Problems setting modules path maps, continuing anyway...
maps__set_modules_path_dir: cannot open /lib/modules/5.11.0-22-generic dir
Problems setting modules path maps, continuing anyway...
Looking at the vmlinux_path (8 entries long)
Failed to open /proc/kcore. Note /proc/kcore requires CAP_SYS_RAWIO capability to access.
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
Using /proc/kallsyms for symbols
Parsing event 'cycles:u'
mmap size 528384B
Reading object code for memory address: 0x7fd8f3dececb
File is: /usr/lib/x86_64-linux-gnu/libc-2.33.so
On file address is: 0x10decb
test child finished with -1
---- end ----
Object code reading: FAILED!
What does the test test for exactly, and why does it fail inside the container? I'm running Docker with --privileged.
Related
Node version: 4.8.0
Platform: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
Node crashed during the Garbace collection but without any other high level pattern (maybe related to https://github.com/nodejs/node/issues/3715).
Unfurtunately I don't have any code to reproduce as I was not able to isolate the problem.
This is the crash stack trace captured with segfault-handler module:
PID 24495 received SIGSEGV for address: 0x3809f3d021f8
<path_node_modules>/segfault-handler/build/Release/segfault-handler.node(+0x1a5b)[0x7f7dd565ca5b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f7dd9c20890]
/usr/bin/nodejs(_ZN2v88internal20MarkCompactCollector22ProcessWeakCollectionsEv+0xfd)[0xaec4dd]
/usr/bin/nodejs(_ZN2v88internal20MarkCompactCollector15MarkLiveObjectsEv+0x214)[0xaf3a14]
/usr/bin/nodejs(_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0x11)[0xaf47e1]
/usr/bin/nodejs(_ZN2v88internal4Heap11MarkCompactEv+0x60)[0xaaafe0]
/usr/bin/nodejs(_ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE+0x4c0)[0xac2be0]
/usr/bin/nodejs(_ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorEPKcS4_NS_15GCCallbackFlagsE+0x238)[0xac30f8]
/usr/bin/nodejs(_ZN2v88internal4Heap15HandleGCRequestEv+0x8f)[0xac3aef]
/usr/bin/nodejs(_ZN2v88internal10StackGuard16HandleInterruptsEv+0x31c)[0xa6041c]
/usr/bin/nodejs(_ZN2v88internal18Runtime_StackGuardEiPPNS0_6ObjectEPNS0_7IsolateE+0x2b)[0xca51ab]
[0x2f2137d0963b]
And also this other stack sometimes:
PID 7545 received SIGSEGV for address: 0x68233500009
/home/documentapp/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1a5b)[0x7f89249bfa5b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f8928f83890]
/usr/bin/nodejs(_ZN2v88internal32IncrementalMarkingMarkingVisitor26VisitFixedArrayIncrementalEPNS0_3MapEPNS0_10HeapObjectE+0x3fe)[0xad51ee]
/usr/bin/nodejs(_ZN2v88internal18IncrementalMarking4StepElNS1_16CompletionActionENS1_18ForceMarkingActionENS1_21ForceCompletionActionE+0x30c)[0xad2a7c]
/usr/bin/nodejs(_ZN2v88internal8NewSpace15SlowAllocateRawEiNS0_19AllocationAlignmentE+0x78)[0xb00f18]
/usr/bin/nodejs(_ZN2v88internal4Heap11AllocateRawEiNS0_15AllocationSpaceES2_NS0_19AllocationAlignmentE+0x109)[0xa64719]
/usr/bin/nodejs(_ZN2v88internal4Heap20AllocateFillerObjectEibNS0_15AllocationSpaceE+0x19)[0xaabd19]
/usr/bin/nodejs(_ZN2v88internal7Factory15NewFillerObjectEibNS0_15AllocationSpaceE+0x2d)[0xa64c5d]
/usr/bin/nodejs(_ZN2v88internal29Runtime_AllocateInTargetSpaceEiPPNS0_6ObjectEPNS0_7IsolateE+0x5e)[0xca52ee]
[0x1e31ede06355]
Can someone give me some hint on how I can find the prblem? Thanks
If you prefer you can also answer on the node issues that I have created:
https://github.com/nodejs/node/issues/11606
Additional information:
Node framework: express, Sails.js
My native modules founded with find node_modules -name '*.node' are:
node_modules/bcrypt/build/Release/bcrypt_lib.node
node_modules/bcrypt/build/Release/obj.target/bcrypt_lib.node
node_modules/segfault-handler/build/Release/segfault-handler.node
node_modules/segfault-handler/build/Release/obj.target/segfault-handler.node
The problems seems to be cause by mongodb logs that fill up the disk space a some point. Was actually hard to see because we clean this periodically so was not critical at the moment I've checked.
Fatal Error -32988: LoadLibrary failed, rc=193 [MsgId: MERR-32988]
Fatal Error -26000: xfbLrwiWebInfraGlobalInitOK failed [MsgId: MERR-26000]
Warning: Extension lrwreplaymain.dll reports error -1 on call to function ExtPerProcessInitialize
Error: Thread Context: Call to service of the driver failed, reason - thread context wasn't initialized on this thread.
Recording is passed but cannot replayed due to the above error.
Thanks for the help in advance.
Either lrwreplaymain.dll or some of the DLLs it depends on were corrupted (error code 193 = ERROR_BAD_EXE_FORMAT). This may have been caused by virus attack, file system corruption etc.
The easiest solution is to re-install LoadRunner. You can also check which DLLs are corrupted by means of the Dependency Walker x86. Just start it and open <LoadRunner installation folder>\bin\lrwreplaymain.dll
7:46:20 PM Gradle sync started
7:46:35 PM Gradle sync failed: Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at https://docs.gradle.org/2.10/userguide/gradle_daemon.html
Please read the following process output to find out more:
-----------------------
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Consult IDE log for more details (Help | Show Log)
the jvm version is 1.7.0_79
and the studio version is 2.1.1
Error occurred during initialization of VM Could not reserve enough space for object heap Error: Could not create the Java Virtual Machine.
There's no space available in RAM. To fix go to /android-studio-dir/bin and edit studio.vmoptions and studio64.vmoptions to increment the -Xmx and to reserve more memory to Java. Note that the number of processes active may influence on that.
Probably, the /tmp location is full..
Found this somewhere..
Use df command
df
You should see an output with a line like this:
tmpfs 102400 102312 88 100% /tmp
So to change the size of the tmp file:
sudo mount -o remount,size=2G /tmp
Done! Now, It should work..
I have a core dump that shows a thread dying from a SIGBUS signal while executing mov %r15d,0xa0(%rsp). That seems to tell me that it died because it ran out of thread stack.
But how can I prove it? I cannot seem to find a GDB command to display thread information besides thread backtraces. In this case there is no backtrace. It shows the current function and then 0x0000000000000000. Yet another indication of stack corruption, I think.
I don't have a copy of /proc/[pid]/maps from when the program died. Is there anything in GDB or in the core file I can look at to find the base of each thread stack?
That seems to tell me that it died because it ran out of thread stack.
Very likely
But how can I prove it?
(gdb) p/x $rsp
$1 = 0x7fffc5791000
(gdb) info target
Symbols from "a.out".
Local core dump file:
`core', file type elf64-x86-64.
0x0000000000400000 - 0x0000000000401000 is load1
...
0x00007faaf2240000 - 0x00007faaf2241000 is load14
0x00007fffc5791000 - 0x00007fffc5f91000 is load15
0x00007fffc5faf000 - 0x00007fffc5fb0000 is load16
0xffffffffff600000 - 0xffffffffff600000 is load17
Local exec file:
...
Note how $rsp is at the (low) end of the load15 segment, and there is no mapping that "covers" $rsp-8
all.
I'm a kernel newbie. I want to know how to get useful infomations from painc such as which line or which function is wrong.
For example, following is a panic-output about usb hiddev, how to read it? Thanks.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
IP: [<ffffffff813b4aa1>] free_async+0xa1/0x100
PGD 2326c9067 PUD 230f4c067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1d.2/usb8/8-2/speed
CPU 3
Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp l]
Pid: 2400, comm: lsusb Tainted: G I--------------- 2.6.32-296.el664fixes.3.x86_64 #1 Dell Inc. OptiPlN
RIP: 0010:[<ffffffff813b4aa1>] do_IRQ: 0.97 No irq handler for vector (irq -1)
[Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 186 is 53003c)
�Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
setfont: KDFONTOP: Invalid argument
Free memory/Total memory (free %): 78672 / 114884 ( 68.4795 )
Loading dm-mod.ko module
Loading dm-log.ko module
Loading dm-region-hash.ko module
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Loading freq_table.ko module
Loading mperf.ko module
Loading ipt_REJECT.ko module
Loading nf_defrag_ipv4.ko module
Loading ip_tables.ko module
Loading nf_conntrack.ko module
Loading ip6_tables.ko module
Loading ipv6.ko module
Loading fat.ko module
Loading macvlan.ko module
Loading tun.ko module
Loading kvm.ko module
Loading uinput.ko module
Loading parport.ko module
Loading dcdbas.ko module
Loading microcode.ko module
The panic itself is actually quite accurate as is;
BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
IP: [] free_async+0xa1/0x100
Tells already that the function where the problem happened is free_async and that function is exactly 0x100 bytes long and the crash happened at offset 0xa1. You need to map that offset into the exact line of code, but that now depends a bit on your environment how to do it.
Sometimes manual code review already will show what line has pointer manipulations, so you can do it just by reviewing that function.
Then the next question is that, why do you have a NULL-pointer there?