This question already has answers here:
How to allocate arrays on the heap in Rust 1.0?
(3 answers)
Thread '<main>' has overflowed its stack when allocating a large array using Box
(2 answers)
Performance comparison of a Vec and a boxed slice
(1 answer)
Closed 4 years ago.
I am trying to initialize a large array with shape [[u64; 4096]; 64]. When I try to initialize this array with [[0; 4096]; 64] I get different results depending on how the program is run.
When I run with cargo test I get the following error:
thread '' has overflowed its stack
fatal runtime error: stack overflow
When I run with either cargo run or cargo test --release my program runs as expected. I think this means that the stack is simply not big enough to handle 8 * 64 * 4096 bytes (just over a MB), and that when I run in release or with cargo run a different sized stack is allocated for the program.
Is my assumption about running out of stack correct?
Could I allocate the array to the heap within a Box instead?
Is this the best option?
I would really like look ups for this array to be as fast as possible.
Once you declare a variable in local scope, it is held on the stack. Since your stack capacity is not enough for the variable you declare, then you get a stack overflow error. I suggest to take a quick look at the book's section on the stack and the heap.
In such big sized objects, declaring them inside a Box makes them stored on the heap, which may be the wiser option for you.
Related
This question already has an answer here:
How to tell if something is heap or stack allocated?
(1 answer)
Closed 2 years ago.
In Rust there are lots of different types; how do I know if something is on the stack vs heap?
Say if I had nested structs 20 levels deep, and every level contains stack data, but at level 20 there is one heap Box, does this mean the entire tree is now on the heap?
And what about if at the root that struct is inside a Box on the heap - is all the stack based data now moved to the heap?
Are there tools to observe this stack vs heap logic?
In Rust there are lots of different types; how do I know if something is on the stack vs heap?
If nobody puts it on the heap, it's not on the heap.
Say if I had nested structs 20 levels deep, and every level contains stack data, but at level 20 there is one heap Box, does this mean the entire tree is now on the heap?
No. Box is specifically a heap pointer. So whatever's inside the box is on the heap. The rest is wherever you put it (the tree could itself be in a box, at which point you'd have a heap-allocated structure pointing to an other heap-allocated structure).
Other common heap pointers are Rc, Arc, String, or Vec. And of course there are other more complicated structures which do their work on the heap without really being pointers e.g. HashMap and friends.
And what about if at the root that struct is inside a Box on the heap - is all the stack based data now moved to the heap?
Yes. Since the rest of the structure is "inside" the "root", you box the entire thing.
Are there tools to observe this stack vs heap logic?
Not really. There are tools like tracing allocators, or debugging support in some allocators (e.g. heap profiling in jemalloc), which let you, well, trace heap allocations. But all these can tell you is that some amount of bytes has been allocated, not what those bytes are exactly, and you'd only know that a heap allocation hasn't happen by deduction that no heap allocation happened.
On Linux, you can profile heap usage with valgrind and massif and view the results with ms_print.
First profile the program:
$ valgrind --tool=massif ./target/debug/my_program
==32006== Massif, a heap profiler
==32006== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==32006== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==32006== Command: ./target/debug/my_program
==32006==
The report gets written to a file named massif.out.<pid> (where pid is the process id) which you can view with ms_print. In the example above the process id is 32006, so view it with:
$ ms_print massif.out.32006
You'll get a lot of information, but it will point you to where heap allocations occur, with the stack traces that lead to them.
In Linux, when a running program attempts to use more stack space than the limit (a stack overflow), that usually results in a "segmentation fault" error and the execution is aborted.
Is it guaranteed that exceeding the stack space limit will always cause a segmentation fault error? Or could it happen that the program continues to run, possibly with some erroneous behavior due to data having been corrupted?
Another way of putting this: if a program misbehaves by producing wrong results but without a crash, can the cause still be a stack overflow?
Edit: to clarify, this question is not about "stack buffer overflow", it is about stack overflow, when the stack space used by the program exceeds the stack size limit (the limit that is in Linux given by ulimit -s).
A stack overflow turning into an access violation requires memory management hardware of some sort. Without hardware-assisted memory protection, an overgrown stack will collide with some other memory allocation, causing mutual corruption.
On demand-paged virtual memory operating systems, the upper limit of the stack is protected by a guard page: a page of virtual memory which is reserved (will not be allocated to anything) and marked "not present" so that accessing it generates a violation. A guard page is only so many bytes wide; a stack pointer can still accidentally increment over the guard page and land in some unrelated writable memory (such as a mapping belonging to a heap allocation) where havoc will be wreaked without necessarily triggering any memory access violation.
In the C language we can easily cause large stack increments by declaring large, uninitialized non-static block-scoped arrays, like char array[8192]; // (twice as large as a 4096 byte guard page). Using features like alloca or C99 variable-length arrays, we can do this dynamically: we can write a program which reads an integer value as a run-time input, and increments the stack by that much.
I debugged a problem many years ago whereby third-party code had debug logging macros, inside whose expansions there was a temporay array like char print_buf[8192] that was used for formatting messages. This was used in a multi-threaded application with many threads, whose stacks were reduced to just 64 kilobytes in size. Thanks to this print_buf, a thread's overflown stack leaped right past the guard page, and landed in another thread's stack, corrupting its local variables, causing the proverbial "hilarity to ensue".
This question already has answers here:
Is there a way to limit the memory, ghci can have?
(2 answers)
Closed 7 years ago.
Here is the problem: sometimes, when playing with GHCI, I end up running an infinite computation by mistake. When this happens, most times, my computer crashes and I'm not even able to interrupt it using Ctrl+C.
I wonder if there is a way to run GHCI (or another interactive console like Hugs) in some mode that allows me to interrupt the program before the memory runs out. Maybe setting a virtual limit for the recursion depth or for the memory used.
(This question may be a duplicated of Is there a way to limit the memory, ghci can have? but this also considers the posibility of a recursion depth limit, not just memory limit.)
You can try using the RTS options to control the garbage collector when starting GHCi. For instance,
ghci +RTS -M100M -RTS Foo.hs
should limit the memory to 100MB. Alternatively, use -Ksize to limit the stack (by default it is limited by 80% of the heap).
I am starting to use asmprofiler for some small programs I am doing as a hobby, now when I am watching the results I see a 'Thread Chart' tab and it shows each thread stack size and stack height vs (time?).
The problem is I don't understand what a thread's stack size and height mean and why this graph could be useful when profiling?
As I read the source code for this program:
Stack height is the number of function call stack frames present on the stack.
Stack size if the size in bytes of the stack.
You might use these graphs if you were:
debugging stack overflows, or
trying to gain understanding of a recursive algorithms performance, or
trying to optimise the reserved stack size for your threads, or
many other reasons that I have not thought of!
I'm working on pthread in android NDK that process video data through network.
I meet problem that is 'stack corruption detected : aborted'. So I set -fstack-check in application.mk, and FATAL SIGNAL 11 blabla.. again.
My conclusion in this problem is about stack size.
When I use window thread, its stack size sets 1kb as default, increase automatically.
but, I don't know about pthread.
Is pthread increase stack size automatically?
p.s. I attached this thread to JavaVM.
On stock Android, pthread stacks are allocated at 1MB by default. (Because of the way the system works, only the parts of the stack that are actually touched get physical pages, so for many threads there's actually only a few KB in use.)
The "stack corruption" message indicates that something has trashed the stack, not that you've run off the end. One way to encounter this is to write off the end of a stack-allocated array. It might be useful to decode the stack trace you get in the log file and see what method it's in when it fails.