I am profiling a Rust binary using perf record and I observe unexpected stack frames in the output of perf script. As a result, the flamegraph is difficult to interpret. How can I avoid these unexpected stack frames?
Profiling commands:
$ perf record -F 997 --call-graph dwarf,65528 -g -e cpu-clock /path/to/bin
$ perf script
In the application, fn main calls fn tip_mine. Here is a record from perf script which shows the expected stack frame:
...
56275bcc4bee stacks_inspect::tip_mine+0xd23e (inlined)
56275bcc4bee stacks_inspect::main+0xd23e (/home/igor/stacks-blockchain/target/release/stacks-inspect)
ffffffffffffffff [unknown] ([unknown])
Here is another record from perf script from the same run which shows extra stack frames between fn main and fn tip_mine:
...
56275bcbe20d stacks_inspect::tip_mine+0x685d (inlined)
56275bcbe20d stacks_inspect::main+0x685d (/home/igor/stacks-blockchain/target/release/stacks-inspect)
56275bc9bc42 core::ops::function::FnOnce::call_once+0x2 (inlined)
56275bc9bc42 std::sys_common::backtrace::__rust_begin_short_backtrace+0x2 (/home/igor/stacks-blockchain/target/release/stacks-inspect)
56275bca5e68 std::rt::lang_start::{{closure}}+0x8 (inlined)
56275c35dc8d core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once+0x42d (inlined)
56275c35dc8d std::panicking::try::do_call+0x42d (inlined)
56275c35dc8d std::panicking::try+0x42d (inlined)
56275c35dc8d std::panic::catch_unwind+0x42d (inlined)
56275c35dc8d std::rt::lang_start_internal::{{closure}}+0x42d (inlined)
56275c35dc8d std::panicking::try::do_call+0x42d (inlined)
56275c35dc8d std::panicking::try+0x42d (inlined)
56275c35dc8d std::panic::catch_unwind+0x42d (inlined)
56275c35dc8d std::rt::lang_start_internal+0x42d (/home/igor/stacks-blockchain/target/release/stacks-inspect)
56275bcc64e7 main+0x27 (/home/igor/stacks-blockchain/target/release/stacks-inspect)
7fa0aa8b4d8f [unknown] (/usr/lib/x86_64-linux-gnu/libc.so.6)
7fa0aa8b4e3f __libc_start_main+0x7f (/usr/lib/x86_64-linux-gnu/libc.so.6)
56275bc5a1d4 _start+0x24 (/home/igor/stacks-blockchain/target/release/stacks-inspect)
Related
I am new to Rust and I try to figure out where things are allocated.
I use Rust 1.64.0 on Ubuntu 22.04.0 (jammy):
$ rustc --version
rustc 1.64.0 (a55dd71d5 2022-09-19)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
I try to explore the executable produce by compiling this very basic Rust program:
fn tester() {
let borrowed_string: &str = "world";
println!("{}", borrowed_string);
}
fn main() { tester(); }
Compilation (OK, this is overkill... but I want to be sure that I have all I need for using GDB):
$ cargo build --config "profile.dev.debug=true" --config "profile.dev.opt-level=0" --profile=dev
Compiling variables v0.1.0 (/home/denis/Documents/github/rust-playground/variables)
Finished dev [unoptimized + debuginfo] target(s) in 0.71s
Run rust-gdb, set the brakpoint and run the program:
$ rust-gdb target/debug/variables
GNU gdb (Ubuntu 12.0.90-0ubuntu1) 12.0.90
...
Reading symbols from target/debug/variables...
(gdb) b 3
Breakpoint 1 at 0x7959: file src/main.rs, line 3.
(gdb) r
Starting program: /home/denis/Documents/github/rust-playground/variables/target/debug/variables
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, variables::tester () at src/main.rs:3
3 println!("{}", borrowed_string);
(gdb)
s
Print the memory mapping;
(gdb) info proc map
process 6985
Mapped address spaces:
Start Addr End Addr Size Offset Perms objfile
0x555555554000 0x55555555a000 0x6000 0x0 r--p /home/denis/Documents/github/rust-playground/variables/target/debug/variables
0x55555555a000 0x555555591000 0x37000 0x6000 r-xp /home/denis/Documents/github/rust-playground/variables/target/debug/variables
0x555555591000 0x55555559f000 0xe000 0x3d000 r--p /home/denis/Documents/github/rust-playground/variables/target/debug/variables
0x5555555a0000 0x5555555a3000 0x3000 0x4b000 r--p /home/denis/Documents/github/rust-playground/variables/target/debug/variables
0x5555555a3000 0x5555555a4000 0x1000 0x4e000 rw-p /home/denis/Documents/github/rust-playground/variables/target/debug/variables
0x5555555a4000 0x5555555c5000 0x21000 0x0 rw-p [heap]
0x7ffff7d5c000 0x7ffff7d5f000 0x3000 0x0 rw-p
0x7ffff7d5f000 0x7ffff7d87000 0x28000 0x0 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7d87000 0x7ffff7f1c000 0x195000 0x28000 r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f1c000 0x7ffff7f74000 0x58000 0x1bd000 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f74000 0x7ffff7f78000 0x4000 0x214000 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f78000 0x7ffff7f7a000 0x2000 0x218000 rw-p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f7a000 0x7ffff7f87000 0xd000 0x0 rw-p
0x7ffff7f87000 0x7ffff7f8a000 0x3000 0x0 r--p /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff7f8a000 0x7ffff7fa1000 0x17000 0x3000 r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff7fa1000 0x7ffff7fa5000 0x4000 0x1a000 r--p /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff7fa5000 0x7ffff7fa6000 0x1000 0x1d000 r--p /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff7fa6000 0x7ffff7fa7000 0x1000 0x1e000 rw-p /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff7fb8000 0x7ffff7fb9000 0x1000 0x0 ---p
0x7ffff7fb9000 0x7ffff7fbd000 0x4000 0x0 rw-p
0x7ffff7fbd000 0x7ffff7fc1000 0x4000 0x0 r--p [vvar]
0x7ffff7fc1000 0x7ffff7fc3000 0x2000 0x0 r-xp [vdso]
0x7ffff7fc3000 0x7ffff7fc5000 0x2000 0x0 r--p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7fc5000 0x7ffff7fef000 0x2a000 0x2000 r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7fef000 0x7ffff7ffa000 0xb000 0x2c000 r--p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7ffb000 0x7ffff7ffd000 0x2000 0x37000 r--p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7ffd000 0x7ffff7fff000 0x2000 0x39000 rw-p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffffffde000 0x7ffffffff000 0x21000 0x0 rw-p [stack]
0xffffffffff600000 0xffffffffff601000 0x1000 0x0 --xp [vsyscall]
(gdb)
We have:
HEAP: [0x5555555A4000, 0x5555555C4FFF]
STACK: [0x7FFFFFFDE000, 0x7FFFFFFFEFFF]
I assume that the data "world" is allocated in the heap. Thus, let's find out... but look at what I obtain!!!
(gdb) find 0x5555555A4000, 0x5555555C4FFF, "world"
0x5555555a4ba0
1 pattern found.
(gdb) find 0x5555555A4000, 0x5555555C4FFF, "world"
0x5555555a4be0
1 pattern found.
(gdb) find 0x5555555A4000, 0x5555555C4FFF, "world"
0x5555555a4c20
1 pattern found.
(gdb) find 0x5555555A4000, 0x5555555C4FFF, "world"
0x5555555a4c60
1 pattern found.
(gdb) find 0x5555555A4000, 0x5555555C4FFF, "world"
0x5555555a4ca0
1 pattern found.
(gdb)
Please note that:
0x5555555A4BA0 -> 0x5555555A4BE0 => 64
0x5555555A4BE0 -> 0x5555555A4C20 => 64
0x5555555A4C20 -> 0x5555555A4C60 => 64
Question: why does the position of the data change from one scan to the next?
(This question may not be related specifically to Rust.)
(gdb) p borrowed_string
$1 = "world"
(gdb) ptype borrowed_string
type = struct &str {
data_ptr: *mut u8,
length: usize,
}
(gdb) p borrowed_string.data_ptr
$2 = (*mut u8) 0x555555591000
(gdb) x/5c 0x555555591000
0x555555591000: 119 'w' 111 'o' 114 'r' 108 'l' 100 'd'
We have:
0x5555555A4000: start of the heap (included)
0x5555555C5000: end of the heap (excluded), and start of the "uncharted" memory territory
0x555555591000: address of the data "world"
0x7FFFF7D5F000: end of the "uncharted" memory territory
Question: what is this "uncharted" memory territory between 0x5555555C5000 (included) and 0x7FFFF7D5F000 (excluded)?
EDIT:
As mentioned in the comments below, I made a mistake. The data "world" is "before" the heap... The comments make sense. Thank you.
Question: what is this "uncharted" memory territory between 0x5555555C5000 (included) and 0x7FFFF7D5F000 (excluded)?
It's nothing. It's memory which is not mapped.
Where are "str" values allocated? Is it in the heap?
Your memory map answers the question:
0x555555591000 0x55555559f000 0xe000 0x3d000 r--p /home/denis/Documents/github/rust-playground/variables/target/debug/variables
This is one of the areas where the binary itself is mapped. Most likely the "rodata" segment of the executable. String literals are generally "allocated" in the binary itself, then a static reference is loaded.
More generally str values are not allocated, they're a reference to data living somewhere else which could be on the heap (if the reference was obtained from a String or Box<str> or some pointers), in the binary (literals, static includes), or even on the stack (e.g. you can create an array on the stack then pass it through std::str::from_utf8 and get an &str out of it)
We have: HEAP: [0x5555555A4000, 0x5555555C4FFF]
That's... not really helpful or useful or true, frankly: modern systems have largely moved away from linear heaps and to memory map heaps. While Linux software still makes some limited use of (s)brk, BSDs qualify them as "historical curiosities". If you assume the heap is contained within these bounds you're going to face lots of surprises.
Question: why does the position of the data change from one scan to the next?
Because the debugger is allocating stuff in context of the program, I assume? So you might be finding the data gdb itself is storing.
I used the following perf command to sample userspace read accesses to DRAM by evince:
perf record -d --call-graph dwarf -c 100 -e mem_load_uops_retired.l3_miss:uppp /opt/evince-3.28.4/bin/evince
As can be seen, I used the PEBS feature to increase the accuracy of sampling. But there are some non-memory accesses reported as memory ones. For example, this is a sampled event reported by perf script:
evince 20589 16079.401401: 100 mem_load_uops_retired.l3_miss:uppp: 555555860750 5080022 N/A|SNP N/A|TLB N/A|LCK N/A
555555579939 ev_history_can_go_back+0x19 (/opt/evince-3.28.4/bin/evince)
5555555862ef ev_window_update_actions_sensitivity+0xa1f (/opt/evince-3.28.4/bin/evince)
55555558ce4f ev_window_page_changed_cb+0xf (/opt/evince-3.28.4/bin/evince)
7ffff574510c g_closure_invoke+0x19c (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff575805d signal_emit_unlocked_R+0xf4d (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff5760714 g_signal_emit_valist+0xa74 (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff576112e g_signal_emit+0x8e (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff7140d76 emit_value_changed+0xf6 (inlined)
7ffff7140d76 adjustment_set_value+0xf6 (inlined)
7ffff7140d76 gtk_adjustment_set_value_internal+0xf6 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff574510c g_closure_invoke+0x19c (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff5757de7 signal_emit_unlocked_R+0xcd7 (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff575fc7f g_signal_emitv+0x27f (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff7153519 gtk_binding_entry_activate+0x289 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff71539ef binding_activate+0x5f (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff7153b7f gtk_bindings_activate_list+0x17f (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff7154cd8 gtk_bindings_activate_event+0x98 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff7973959 ev_view_key_press_event+0x59 (/opt/evince-3.28.4/lib/libevview3.so.3.0.0)
7ffff72698f6 _gtk_marshal_BOOLEAN__BOXEDv+0xa6 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff574524f _g_closure_invoke_va+0xbf (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff57603cc g_signal_emit_valist+0x72c (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff576112e g_signal_emit+0x8e (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff73b1533 gtk_widget_event_internal+0x163 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff73d1f0a gtk_window_propagate_key_event+0xfa (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
5555555894b1 ev_window_key_press_event+0x31 (/opt/evince-3.28.4/bin/evince)
7ffff72698f6 _gtk_marshal_BOOLEAN__BOXEDv+0xa6 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff5745345 _g_closure_invoke_va+0x1b5 (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff57603cc g_signal_emit_valist+0x72c (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff576112e g_signal_emit+0x8e (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff73b1533 gtk_widget_event_internal+0x163 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff726693e propagate_event+0x21e (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff7268947 gtk_main_do_event+0x7f7 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff6d79764 _gdk_event_emit+0x24 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6da9f91 gdk_event_source_dispatch+0x21 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff546a416 g_main_dispatch+0x2e6 (inlined)
7ffff546a416 g_main_context_dispatch+0x2e6 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
7ffff546a64f g_main_context_iterate+0x1ff (inlined)
7ffff546a6db g_main_context_iteration+0x2b (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
7ffff5a2be3c g_application_run+0x1fc (/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.5600.4)
555555573707 main+0x447 (/opt/evince-3.28.4/bin/evince)
7ffff4a91b96 __libc_start_main+0xe6 (/lib/x86_64-linux-gnu/libc-2.27.so)
555555573899 _start+0x29 (/opt/evince-3.28.4/bin/evince)
ffffffffffffffff [unknown] ([unknown])
This implies that there exists an access to address 0x555555860750 (which is located in [heap]) by an instruction at 0x555555579939 (which is located in evince text section at offset 0x19 of function ev_history_can_go_back()). This memory instrucion is the last line in the following code snippet:
0000000000025920 <ev_history_can_go_back>:
25920: 53 push %rbx
25921: 48 89 fb mov %rdi,%rbx
25924: e8 67 fa ff ff callq 25390 <ev_history_get_type>
25929: 48 85 db test %rbx,%rbx
2592c: 74 42 je 25970 <ev_history_can_go_back+0x50>
2592e: 48 8b 13 mov (%rbx),%rdx
25931: 48 85 d2 test %rdx,%rdx
25934: 74 05 je 2593b <ev_history_can_go_back+0x1b>
25936: 48 39 02 cmp %rax,(%rdx)
25939: 74 0f je 2594a <ev_history_can_go_back+0x2a>
This is a jump to ev_history_can_go_back+0x2a and, apparently, this is not an access to [heap] at address 0x555555860750. Is this perf report wrong?
UPDATE
How about the following backtrace?
11159097179866 0xfb80 [0x1778]: PERF_RECORD_SAMPLE(IP, 0x4002): 7309/7309: 0x7ffff6d6c310 period: 10000 addr: 0x7ffff7034e50
... FP chain: nr:0
... user regs: mask 0xff0fff ABI 64-bit
.... AX 0x555555b8b4c0
.... BX 0x555555c48e10
.... CX 0x1
.... DX 0x7fffffffd988
.... SI 0x7fffffffd980
.... DI 0x555555b8b4c0
.... BP 0x258
.... SP 0x7fffffffd978
.... IP 0x7ffff6d6c310
.... FLAGS 0x20e
.... CS 0x33
.... SS 0x2b
.... R8 0x27c
.... R9 0x24
.... R10 0x2a2
.... R11 0x0
.... R12 0x258
.... R13 0x555555b8b4c0
.... R14 0x3000
.... R15 0x7ffff5747000
... ustack: size 5768, offset 0xd8
. data_src: 0x5080022
... thread: evince:7309
...... dso: /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30
evince 7309 11159.097179: 10000 mem_load_uops_retired.l3_miss:uppp: 7ffff7034e50 5080022 N/A|SNP N/A|TLB N/A|LCK N/A
7ffff6d6c310 cairo_surface_get_device_scale#plt+0x0 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d91029 gdk_window_create_similar_surface+0xc9 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d95410 gdk_window_begin_paint_internal+0x350 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d956f1 gdk_window_begin_draw_frame+0xc1 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff73c4942 gtk_widget_render+0xd2 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff7268858 gtk_main_do_event+0x708 (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2200.30)
7ffff6d79764 _gdk_event_emit+0x24 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d897f4 _gdk_window_process_updates_recurse_helper+0x104 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d8a9f5 gdk_window_process_updates_internal+0x165 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d8abef gdk_window_process_updates_with_mode+0x11f (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff574510c g_closure_invoke+0x19c (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff575805d signal_emit_unlocked_R+0xf4d (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff5760714 g_signal_emit_valist+0xa74 (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff576112e g_signal_emit+0x8e (/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.5600.4)
7ffff6d82ac8 gdk_frame_clock_paint_idle+0x3c8 (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff6d6e07f gdk_threads_dispatch+0x1f (/usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2200.30)
7ffff546ad02 g_timeout_dispatch+0x12 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
7ffff546a284 g_main_dispatch+0x154 (inlined)
7ffff546a284 g_main_context_dispatch+0x154 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
7ffff546a64f g_main_context_iterate+0x1ff (inlined)
7ffff546a6db g_main_context_iteration+0x2b (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
7ffff5a2be3c g_application_run+0x1fc (/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.5600.4)
555555573707 main+0x447 (/opt/evince-3.28.4/bin/evince)
7ffff4a91b96 __libc_start_main+0xe6 (/lib/x86_64-linux-gnu/libc-2.27.so)
555555573899 _start+0x29 (/opt/evince-3.28.4/bin/evince)
The access point is at offset 0 of the following disassembly:
Dump of assembler code for function cairo_surface_get_device_scale#plt:
0x000000000002a310 <+0>: jmpq *0x2c8b3a(%rip) # 0x2f2e50
0x000000000002a316 <+6>: pushq $0x1c7
0x000000000002a31b <+11>: jmpq 0x28690
This is an unconditional jump which will not lead to macrofusion.
On Intel CPUs at least, cmp %rax,(%rdx) can macro-fuse with the following je, while also micro-fusing the load. https://agner.org/optimize/. Also related: Micro fusion and addressing modes (this is a non-indexed addressing mode so this can stay micro-fused even on Sandybridge/IvyBridge).
So in the fused domain (where retirement happens) you really do have single-uop compare-and-branch with a memory source. Note that mem_load_uops_retired.l3_miss:uppp counts uops, not instructions.
Even in the unfused domain, macro-fused compare/branch really executes as a single uop on a single execution unit, but the load does have to execute on a separate port. (Micro-fusion saves decode/issue front-end bandwidth, and uop cache space, but not back-end ports.)
I was running Gapcloser (a tool released from Soapdenovo), and it kept comes up a message and failed in the middle of the process:
*** Error in `GapCloser': free(): invalid pointer: 0x0000000001eb4a80 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f7658b0a7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f7658b1337a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f7658b1753c]
GapCloser[0x405d41]
GapCloser[0x40983b]
GapCloser[0x405ad3]
GapCloser[0x403982]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f7658ab3830]
GapCloser[0x402b59]
======= Memory map: ========
00400000-0042a000 r-xp 00000000 fc:02 93192245 /opt/GapCloser-v1.12-r6/Debug/GapCloser
00629000-0062a000 r--p 00029000 fc:02 93192245 /opt/GapCloser-v1.12-r6/Debug/GapCloser
0062a000-0062b000 rw-p 0002a000 fc:02 93192245 /opt/GapCloser-v1.12-r6/Debug/GapCloser
01e9b000-01ed7000 rw-p 00000000 00:00 0 [heap]
7f6689a9b000-7f6aeab13000 rw-p 00000000 00:00 0
7f6f2ac77000-7f6f3c4b9000 rw-p 00000000 00:00 0
7f6f44000000-7f6f44021000 rw-p 00000000 00:00 0
7f6f44021000-7f6f48000000 ---p 00000000 00:00 0
7f6f4b9f4000-7f7658a93000 rw-p 00000000 00:00 0
7f7658a93000-7f7658c53000 r-xp 00000000 fc:00 66322933 /lib/x86_64-linux-gnu/libc-2.23.so
7f7658c53000-7f7658e53000 ---p 001c0000 fc:00 66322933 /lib/x86_64-linux-gnu/libc-2.23.so
7f7658e53000-7f7658e57000 r--p 001c0000 fc:00 66322933 /lib/x86_64-linux-gnu/libc-2.23.so
7f7658e57000-7f7658e59000 rw-p 001c4000 fc:00 66322933 /lib/x86_64-linux-gnu/libc-2.23.so
7f7658e59000-7f7658e5d000 rw-p 00000000 00:00 0
7f7658e5d000-7f7658e74000 r-xp 00000000 fc:00 66326925 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f7658e74000-7f7659073000 ---p 00017000 fc:00 66326925 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f7659073000-7f7659074000 r--p 00016000 fc:00 66326925 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f7659074000-7f7659075000 rw-p 00017000 fc:00 66326925 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f7659075000-7f765917d000 r-xp 00000000 fc:00 66322974 /lib/x86_64-linux-gnu/libm-2.23.so
7f765917d000-7f765937c000 ---p 00108000 fc:00 66322974 /lib/x86_64-linux-gnu/libm-2.23.so
7f765937c000-7f765937d000 r--p 00107000 fc:00 66322974 /lib/x86_64-linux-gnu/libm-2.23.so
7f765937d000-7f765937e000 rw-p 00108000 fc:00 66322974 /lib/x86_64-linux-gnu/libm-2.23.so
7f765937e000-7f76594fa000 r-xp 00000000 fc:00 51382039 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
7f76594fa000-7f76596fa000 ---p 0017c000 fc:00 51382039 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
7f76596fa000-7f7659704000 r--p 0017c000 fc:00 51382039 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
7f7659704000-7f7659706000 rw-p 00186000 fc:00 51382039 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
7f7659706000-7f765970a000 rw-p 00000000 00:00 0
7f765970a000-7f7659722000 r-xp 00000000 fc:00 66323020 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f7659722000-7f7659921000 ---p 00018000 fc:00 66323020 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f7659921000-7f7659922000 r--p 00017000 fc:00 66323020 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f7659922000-7f7659923000 rw-p 00018000 fc:00 66323020 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f7659923000-7f7659927000 rw-p 00000000 00:00 0
7f7659927000-7f765994d000 r-xp 00000000 fc:00 66322909 /lib/x86_64-linux-gnu/ld-2.23.so
7f7659afc000-7f7659b3a000 rw-p 00000000 00:00 0
7f7659b4b000-7f7659b4c000 rw-p 00000000 00:00 0
7f7659b4c000-7f7659b4d000 r--p 00025000 fc:00 66322909 /lib/x86_64-linux-gnu/ld-2.23.so
7f7659b4d000-7f7659b4e000 rw-p 00026000 fc:00 66322909 /lib/x86_64-linux-gnu/ld-2.23.so
7f7659b4e000-7f7659b4f000 rw-p 00000000 00:00 0
7ffc5a9e7000-7ffc5aa08000 rw-p 00000000 00:00 0 [stack]
7ffc5abaa000-7ffc5abad000 r--p 00000000 00:00 0 [vvar]
7ffc5abad000-7ffc5abaf000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
And this is my log file:
cat gapcloser.log
[WARNING] Maximum supported read length is 155 bp! Program will use 155 instead of 251.
Program: GapCloser
Version: 1.12
Parameters:
-a (scaffold file): sunbird_platanus_raw_kraken_scaff_scaffold.fa
-b (config file): sunbird_raw.config
-o (output file): sunbird_platanus_raw_kraken_scaff_gapclosed
-l (max read len): 155
-p (overlap para): 25
-t (thread num): 40
>>>>>>>>>>hash initializing<<<<<<<<<<
counting reads
in lib:
total reads sum: 289413008
Paired-end reads sum: 289413008
in lib:
total reads sum: 21371568
Paired-end reads sum: 21371568
in lib:
total reads sum: 21944342
Paired-end reads sum: 21944342
in lib:
total reads sum: 20671814
Paired-end reads sum: 20671814
in lib:
total reads sum: 20949232
Paired-end reads sum: 20949232
in lib:
total reads sum: 22977390
Paired-end reads sum: 22977390
in lib:
total reads sum: 19285920
Paired-end reads sum: 19285920
in lib:
total reads sum: 23065494
Paired-end reads sum: 23065494
in lib:
total reads sum: 21131136
Paired-end reads sum: 21131136
in lib:
total reads sum: 20350118
Paired-end reads sum: 20350118
in lib:
total reads sum: 20579926
Paired-end reads sum: 20579926
in lib:
total reads sum: 18998014
Paired-end reads sum: 18998014
in lib:
total reads sum: 20039196
Paired-end reads sum: 20039196
reads sum: 540777158
spent time: 1658s
reads counted
allocating memorys
spent time: 19s
memorys allocated
initializing reads
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
in lib:
spent time: 1984s
reads initialized
sorting reads in reversed direction
spent time: 1357s
reads sorted
initializing hash on reversed reads
kmers sum: 413463339
spent time: 951s
hash initialized
reversing reads
spent time: 633s
reads reversed
sorting reads in forward direction
spent time: 1363s
reads sorted
initializing hash on forward reads
kmers sum: 783012674
spent time: 994s
hash initialized
marking unique reads
spent time: 41s
unique reads marked
sorting reads by serial number
spent time: 1011s
reads sorted
spent total time: 10011s
>>>>>>>>>>hash initialization finished<<<<<<<<<<
In fact, I have run this programme successfully with the same set of data before, however our server has been re-installed therefore Gapcloser was re-installed again, but running the same data no longer works due to this issue...
I thought it was memory's problem so I cleaned the memory page cache by this command, but it did not help at all, this is my current memory, which should definitely be enough for Gapcloser.
free -h
total used free shared buff/cache available
Mem: 755G 2.7G 539G 14M 214G 751G
Swap: 810G 106M 810G
By the way I am using Gapcloser source code version, and my ubuntu is 16.04 version.
This is my Soapdenovo/Gapcloser Config file which I think it's not a problem because it seems to have read all the libraries successfully?
#FOR MATEPAIR LIBRARIES
#maximal read length
max_rd_len=251
#average insert size for 1kb
[LIB]
#average insert size for gel free
avg_ins=1805
#reverse-forward, generated from circularizing libraries with typical insert size greater than 2Kb
reverse_seq=1
#In which parts the reads are used. 2 - only scaffold assembly
asm_flags=4
#assembler will cut the reads from the current library to this length
rd_len_cutof=150
#In which order the reads are used while scaffolding,1-170/200/250bp,2-350/500bp,3-800bp,4-2KB,5-5Kb,6-10Kb,7-20Kb
rank=4
#fastq files for 1kb1
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads1K1_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads1K1_R2.fastq
[LIB]
#average insert size for 3kb1
avg_ins=3918
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=5
#fastq files for 3kb
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads3K1_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads3K1_R2.fastq
[LIB]
#average insert size for 5kb1
avg_ins=6243
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=5
#fastq files for 5kb
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads5K1_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads5K1_R2.fastq
[LIB]
#average insert size for 8kb1
avg_ins=8930
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=6
#fastq files for 8kb
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads8K1_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads8K1_R2.fastq
[LIB]
#average insert size for 12kb1
avg_ins=12507
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=6
#fastq files for 12kb
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads12K1_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads12K1_R2.fastq
[LIB]
#average insert size for 20kb1
avg_ins=17407
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=7
#fastq files for 20kb
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads20K1_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads20K1_R2.fastq
[LIB]
#average insert size for 1kb2
avg_ins=9793
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=4
#fastq files for 1kb2
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads1K2_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads1K2_R2.fastq
[LIB]
#average insert size for 3kb2
avg_ins=3944
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=5
#fastq files for 3kb2
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads3K2_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads3K2_R2.fastq
[LIB]
#average insert size for 5kb2
avg_ins=6225
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=5
#fastq files for 5kb2
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads5K2_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads5K2_R2.fastq
[LIB]
#average insert size for 8kb2
avg_ins=8913
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=6
#fastq files for 8kb2
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads8K2_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads8K2_R2.fastq
[LIB]
#average insert size for 12kb2
avg_ins=12491
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=6
#fastq files for 12kb2
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads12K2_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads12K2_R2.fastq
[LIB]
#average insert size for 20kb2
avg_ins=17370
reverse_seq=1
asm_flags=4
rd_len_cutoff=150
rank=7
#fastq files for 20kb2
q1=/home/cch/sunbird/sunbird_kraken/Pure_MPreads20K2_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_MPreads20K2_R2.fastq
### 2x250 run data
[LIB]
avg_ins=434
reverse_seq=0
#only contigs assembly
asm_flags=4
rank=2
#fastq files from LGC16_JM06_591bp
q1=/home/cch/sunbird/sunbird_kraken/Pure_LGC16_JM09_594bp_CTTGTA_L002_R1.fastq
q2=/home/cch/sunbird/sunbird_kraken/Pure_LGC16_JM09_594bp_CTTGTA_L002_R2.fastq
I have run the Gapcloser through Valgrind and got 2 errors, but I do not know how to fix it, does anyone have any ideas?
valgrind --leak-check=yes GapCloser
==77240== Memcheck, a memory error detector
==77240== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==77240== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==77240== Command: GapCloser
==77240==
==77240== Syscall param open(filename) points to unaddressable byte(s)
==77240== at 0x59FB040: __open_nocancel (syscall-template.S:84)
==77240== by 0x597DACD: _IO_file_open (fileops.c:221)
==77240== by 0x597DD34: _IO_file_fopen##GLIBC_2.2.5 (fileops.c:328)
==77240== by 0x5971D33: __fopen_internal (iofopen.c:86)
==77240== by 0x510AD2F: std::__basic_file<char>::open(char const*, std::_Ios_Openmode, int) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
==77240== by 0x514B259: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
==77240== by 0x514B42F: std::basic_ifstream<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
==77240== by 0x403450: main (Main.cpp:121)
==77240== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==77240==
[Error] Can not open input file.
Version:
1.12
Contact:
soap#genomics.org.cn
Usage:
GapCloser [options]
-a <string> input scaffold file name, required.
-b <string> input library info file name, required.
-o <string> output file name, required.
-l <int> maximum read length (<=155), default=100.
-p <int> overlap param(<=31), default=25.
-t <int> thread number, default=1.
-h -? output help information.
==77240==
==77240== Process terminating with default action of signal 27 (SIGPROF)
==77240== at 0x5A0CE0F: write_gmon (gmon.c:354)
==77240== by 0x5A0D589: _mcleanup (gmon.c:422)
==77240== by 0x593DFF7: __run_exit_handlers (exit.c:82)
==77240== by 0x593E044: exit (exit.c:104)
==77240== by 0x40315F: usage() (Main.cpp:60)
==77240== by 0x403497: main (Main.cpp:124)
==77240==
==77240== HEAP SUMMARY:
==77240== in use at exit: 299,124 bytes in 2 blocks
==77240== total heap usage: 4 allocs, 2 frees, 300,700 bytes allocated
==77240==
==77240== LEAK SUMMARY:
==77240== definitely lost: 0 bytes in 0 blocks
==77240== indirectly lost: 0 bytes in 0 blocks
==77240== possibly lost: 0 bytes in 0 blocks
==77240== still reachable: 299,124 bytes in 2 blocks
==77240== suppressed: 0 bytes in 0 blocks
==77240== Reachable blocks (those to which a pointer was found) are not shown.
==77240== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==77240==
==77240== For counts of detected and suppressed errors, rerun with: -v
==77240== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 0 from 0)
Profiling timer expired
While I was working on a bit of C code, I came across this strange bug.
I made a mistake in my code and wrote to buf rather than &buf, but it worked almost just fine.
...
void* buf;
int ret;
int fd = open("1", O_CREAT | O_RDWR, 0777);
write(fd, "test\n", 5);
lseek(fd, 0, SEEK_SET);
ret = read(fd, buf, 5); // Yes, this should be &buf
printf("Ret: %d Str: %s\n", ret, buf);
---- output ----
Ret: 5 Str: test\n
This code works and I get test\n in my stdout, even though I should have had &buf in my read call. Please, I am aware that changing buf to &buf works. That is not the question.
This is what does not work:
...
void* buf;
void* blah = "a"; // Using char* still did not work
int ret;
int fd = open("1", O_CREAT | O_RDWR, 0777);
write(fd, "test\n", 5);
lseek(fd, 0, SEEK_SET);
ret = read(fd, buf, 5);
printf("Ret: %d Str: %s\n", ret, buf);
---- output ----
Ret: -1 Str: 1�I��^H��H���PTI��`#
The binary for file 1 is the same for both programs. No error in writing to 1.
Why does the first code snippet work?
How does adding a variable that is never used make this no longer
work?
Why did writing to buf and not &buf work in the first place?
Here is the strings section in each binary:
Functioning code:
0000770: 0100 0200 0000 0000 0000 0000 0000 0000 ................
0000780: 3100 7465 7374 0a00 4572 723a 2025 640a 1.test..Err: %d.
0000790: 0a00 5374 723a 2025 730a 0000 011b 033b ..Str: %s......;
00007a0: 3000 0000 0500 0000 34fd ffff 7c00 0000 0.......4...|...
Malfunctioning code:
0000770: 0100 0200 0000 0000 0000 0000 0000 0000 ................
0000780: 6100 3100 7465 7374 0a00 4572 723a 2025 a.1.test..Err: %
0000790: 640a 0a00 5374 723a 2025 730a 0000 0000 d...Str: %s.....
00007a0: 011b 033b 3400 0000 0500 0000 30fd ffff ...;4.......0...
Thanks.
(Since it's pretty much impossible to put more than a line of code in comments)
Warnings from compiling with -Wall -Wextra:
x.c: In function ‘main’:
x.c:15:25: warning: format ‘%s’ expects argument of type ‘char *’, but argument 3 has type ‘void *’ [-Wformat=]
printf("Ret: %d Str: %s\n", ret, buf);
~^
%p
x.c:14:9: warning: ‘buf’ is used uninitialized in this function [-Wuninitialized]
ret = read(fd, buf, 5);
^~~~~~~~~~~~~~~~
The results of running your program through valgrind:
==6978== Memcheck, a memory error detector
==6978== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6978== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==6978== Command: ./a.out
==6978==
==6978== Syscall param read(buf) contains uninitialised byte(s)
==6978== at 0x4F4C081: read (read.c:27)
==6978== by 0x1087BF: main (x.c:14)
==6978==
==6978== Syscall param read(buf) points to unaddressable byte(s)
==6978== at 0x4F4C081: read (read.c:27)
==6978== by 0x1087BF: main (x.c:14)
==6978== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==6978==
==6978== Conditional jump or move depends on uninitialised value(s)
==6978== at 0x4E97A41: vfprintf (vfprintf.c:1643)
==6978== by 0x4EA0F25: printf (printf.c:33)
==6978== by 0x1087DC: main (x.c:18)
==6978==
Ret: -1 Str: (null)
==6978==
==6978== HEAP SUMMARY:
==6978== in use at exit: 0 bytes in 0 blocks
==6978== total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated
==6978==
==6978== All heap blocks were freed -- no leaks are possible
==6978==
==6978== For counts of detected and suppressed errors, rerun with: -v
==6978== Use --track-origins=yes to see where uninitialised values come from
==6978== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)
You have to make buf point to valid memory you have permission to use or you get undefined behavior where anything can happen. If you're lucky, it'll just crash your program, but you can't count on that.
Heres my code:
factorial :: Integer -> Integer
factorial n = product [1..n]
main = print(factorial 50)
I don't get any errors compiling, but when i run the compiled code
runhaskell test
I get this error:
test:1:1: lexical error at character '\DEL'
What is causing this? How do I solve the problem?
UPDATES
I did a hexdump of the file:
$ hexdump -x test.hs
and got
0000000 6166 7463 726f 6169 206c 3a3a 4920 746e
0000010 6765 7265 2d20 203e 6e49 6574 6567 0a72
0000020 6166 7463 726f 6169 206c 206e 203d 7270
0000030 646f 6375 2074 315b 2e2e 5d6e 6d0a 6961
0000040 206e 203d 7270 6e69 2874 6166 7463 726f
0000050 6169 206c 3035 0029
0000057
Make sure that you're using runhaskell with the source file test.hs rather than the compiled binary test.
If you've used something like ghc to create an executable file, you can just run that directly, with something like:
./test
Be aware that test is probably not a good name for an executable since it's a built-in command on some shells, something that's burnt me before when my test executable doesn't seem to do what I wanted :-)