I recently ran into a bug with the "linux stack" and the "linux stack size". I came across a blog directing me to try
ulimit -a
to see what the limit for my box was, and it was set to 8192kb which seems to be the default.
What is the "linux stack"? How does it work, what does it store, what does it do?
The short answer is:
When programs on your linux box run, they add and remove data from the stack on a regular basis as the programs function. The stack size, referes to how much space is allocated in memory for the stack. If you increase the stack size, that allows the program to increase the number of routines that can be called. Each time a function is called, data can be added to the stack (stacked on top of the last routines data.)
Unless the program is a very complex, or designed for a special purpose, a stack size of 8192kb is normally fine. Some programs like graphics processing programs require you to increase the size of the stack to function. As they may store a lot of data on the stack.
Feel free to increase the stack size for those applications, its not a problem. To do so, use
ulimit -s bytes
BTW, What is a StackOverflowError?
Related
I'm doing computations with huge arrays and for some of this computations I need an increased stack size! Is there any downside of setting the stack size to unlimited (ulimit -s unlimited) in my ~/.bashrc?
The program is written in fortran(F77 & F90) and parallelized with MPI. Some of my arrays have more than 2E7 entries and when I use a small number of cores with MPI it crashes with segmentation fault.
The array size stays the same through the whole computation therefore I setted them to fixes value:
real :: p(200,200,400)
integer :: ib,ie,jb,je,kb,ke
...
ib=1;ie=199
jb=2;je=198
kb=2;ke=398
call SOLVE_POI_EQ(rank,p(ib:ie,jb:je,kb:ke),R)
Setting the stacksize to unlimited likely won't help you. You are allocating a chunk of 64MB on the stack, and likely don't fill it from the top, but from the bottom.
This is important, because the OS grows the stack as you go. Whenever it detects a page-fault right below the stack segment, it will assume that you need more space, and silently insert a new page. The size of this trigger-region within your address-space is limited, though, and I doubt that its larger than 64 MB. Since you index variables are likely placed below your array on the stack, accessing them already does the 64 MB jump that kills your process.
Just make your array allocatable, add the corresponding allocate() statement, and you should be fine.
Stack size is never really unlimited, so you would still have some failures. And your code still won't be portable to Linux systems with smaller (or normal-sized) stacks.
BTW, you should explain which kind of programs are you running, show some source code.
If coding in C++, using standard containers should help a lot (regarding actual stack consumption). For example, a local (stack allocated) std::vector<int> v(10000); (instead of int v[10000];) has its data allocated on the heap (and deallocated by the destructor when you exit from the block defining it)
It would be much better to improve your programs to avoid excessive stack consumption. The need of a lot of stack space is really a bug that you should try to correct. A typical rule of thumb is to have call frames smaller than a few kilobytes (so allocate any larger data on the heap).
You might consider also using the Boehm conservative garbage collector: you would use GC_MALLOC instead of malloc (and you would heap allocate large data structure using GC_MALLOC) but you won't have to bother to free your (GC-heap allcoated) data.
I have a few questions related to limitations on the stack size for Linux. I'm most interested in x86_64 systems, but if there are platform differences I'd like to understand them as well. My questions are:
1) How does Linux dynamically increase the size of the stack?
I've written a test program with a recursive function (to use stack space) where I can specify the number of iterations as a command line parameter. The program pauses waiting for user input after finishing the recursion, which allows me to get information about the running process. If I run with a small number of iterations and then use pmap to view the stack size it is 132K.
00007fff0aa3c000 132K rw--- [ stack ]
Whereas if I run with a larger number of iterations the size can grow much larger, I believe up to 8192 KB by default. For example here's output from running with more iterations.
00007fff3ed75000 8040K rw--- [ stack ]
But if I use strace to trace the system calls when running the application I don't see any associated with growing the stack. So I'm wondering what the kernel is doing to manage the stack space for a process.
2) Does Linux protect the stack region in any way when setting ulimit -s unlimited?
If I use the command ulimit -s unlimited then I'm able to run many more iterations of my recursive function and the stack grows much larger. For example here's output from pmap
00007ffda43a3000 8031260K rw--- [ stack ]
Since I don't want to cause my machine to crash/hang/lockup I haven't tested with infinite recursion yet. But I'm wondering if I did is there anything that would cause the kernel to detect stack overflow. Or is ulimit the only protection and turning that off allows the stack to grow unbounded?
3) How are stack guard pages handled?
This isn't directly related to anything I've experimented with, but I'm also wondering how Linux manages stack guard pages in conjunction with allowing the stack to grow dynamically.
For each running process, Linux keeps a list of regions of the virtual memory addresses. If an address reference generates a page fault, Linux checks that list to see if the virtual address is legal (within the range of one of the regions). If not claimed by a region, the application gets a SIGSEGV error, otherwise the kernel allocates another page of system memory and adds to the translation caches. if the faulting address just misses a region, and that region is for a stack (which grow up or down, according to the machine architecture) then Linux allocates another VM page, maps it into the region, thus growing the stack.
The kernel does not protect the stack. If a stack access causes a page fault because a physical VM page is not attached to a memory region for the process, the process's rlimit is tested to see if adding another page is permitted.
Stack guard pages are using by some malloc(3) debugger libraries. What these to is to expand each memory request by 2 VM pages: one page before the new page, one page after it. The extra pages are marked as no-access-at-all so if the application walks off the end of a region, or moves before the beginning, the application gets an access violation.
The above has been shamelessly over-simplified but still should give the gist.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 8 years ago.
Improve this question
My Tcl script aborts, saying that it is unable to realloc 2191392 bytes. This happens when the script is kept for a longer execution duration, say, more than 10 hours. My Tcl script connects to devices using telnet and ssh connections and executes/verifies some command outputs on devices. The Linux machine has enough RAM 32GB, and ulimit is unlimited for process, data, file size. My script process doesn't eat up more memory, but the worst case is < 1GB. I just wonder why memory allocation is failed having plenty of RAM.
That message is an indication that an underlying malloc() call returned NULL (in a non-recoverable location), and given that it's not for very much, it's an indication that the system is thoroughly unable to allocate much memory. Depending on how your system is configured (32-bit vs. 64-bit; check what parray tcl_platform prints to find out) that could be an artefact of a few things, but if you think that it shouldn't be using more than a gigabyte of memory, it's an indication of a memory leak.
Unfortunately, it's hard to chase down memory leaks in general. Tcl's built-in memory command (enabled via configure --enable-symbols=mem at build time) can help, as can a tool like Electric Fence, but they are imperfect and can't generally tell you where you're getting things wrong (as you'll be looking for the absence of something to release memory). At the Tcl level, see whether each of the variables listed by info globals is of sensible size or whether there's a growing number of globals. You'll want to use tools like string length, array exists, array size and array names for this.
It's also possible for memory allocation to fail due to another process consuming so much memory that the OS starts to feel highly constrained. I hope this isn't happening in your case, since it's much harder to prevent.
I'm looking for a good description of stacks within the linux kernel, but I'm finding it surprisingly difficult to find anything useful.
I know that stacks are limited to 4k for most systems, and 8k for others. I'm assuming that each kernel thread / bottom half has its own stack. I've also heard that if an interrupt goes off, it uses the current thread's stack, but I can't find any documentation on any of this. What I'm looking for is how the stacks are allocated, if there's any good debugging routines for them (I'm suspecting a stack overflow for a particular problem, and I'd like to know if its possible to compile the kernel to police stack sizes, etc).
The reason that documentation is scarce is that it's an area that's quite architecture-dependent. The code is really the best documentation - for example, the THREAD_SIZE macro defines the (architecture-dependent) per-thread kernel stack size.
The stacks are allocated in alloc_thread_stack_node(). The stack pointer in the struct task_struct is updated in dup_task_struct(), which is called as part of cloning a thread.
The kernel does check for kernel stack overflows, by placing a canary value STACK_END_MAGIC at the end of the stack. In the page fault handler, if a fault in kernel space occurs this canary is checked - see for example the x86 fault handler which prints the message Thread overran stack, or stack corrupted after the Oops message if the stack canary has been clobbered.
Of course this won't trigger on all stack overruns, only the ones that clobber the stack canary. However, you should always be able to tell from the Oops output if you've suffered a stack overrun - that's the case if the stack pointer is below task->stack.
You can determine the process stack size with the ulimit command. I get 8192 KiB on my system:
$ ulimit -s
8192
For processes, you can control the stack size of processes via ulimit command (-s option). For threads, the default stack size varies a lot, but you can control it via a call to pthread_attr_setstacksize() (assuming you are using pthreads).
As for the interrupt using the userland stack, I somewhat doubt it, as accessing userland memory is a kind of a hassle from the kernel, especially from an interrupt routine. But I don't know for sure.
I am developing a multithread modular application using C programming language and NPTL 2.6. For each plugin, a POSIX thread is created. The problem is each thread has its own stack area, since default stack size depends on user's choice, this may results in huge memory consumption in some cases.
To prevent unnecessary memory usage I used something similar to this to change stack size before creating each thread:
pthread_attr_t attr;
pthread_attr_init (&attr);
pthread_attr_getstacksize(&attr, &st1);
if(pthread_attr_setstacksize (&attr, MODULE_THREAD_SIZE) != 0) perror("Stack ERR");
pthread_attr_getstacksize(&attr, &st2);
printf("OLD:%d, NEW:%d - MIN: %d\n", st1, st2, PTHREAD_STACK_MIN);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
/* "this" is static data structure that stores plugin related data */
pthread_create(&this->runner, &attr, (void *)(void *)this->run, NULL);
EDIT I: pthread_create() section added.
This did not work work as I expected, the stack size reported by pthread_attr_getstacksize() is changed but total memory usage of the application (from ps/top/pmap output) did not changed:
OLD:10485760, NEW:65536 - MIN: 16384
When I use ulimit -s MY_STACK_SIZE_LIMIT before starting application I achieve the expected result.
My questions are:
1-) Is there any portable(between UNIX variants) way to change (default)thread stack size after starting application(before creating thread of course)?
2-) Is it possible to use same stack area for every thread?
3-) Is it possible completely disable stack for threads without much pain?
Answers for #2 and #3 are no and no. Each thread needs a stack (where else do your local variables and return addresses go?) and they need to be unique per-thread (otherwise threads would overwrite each other's local variables and return addresses, making everybody crash).
As for #1... the set stack size call is precisely the answer for this. I suggest you figure out an acceptable size to create your threads with, and set it.
As for why things don't look right to you in top.... top is a notorious liar about memory usage. :-) Is stuff actually failing to be allocated or getting OOM-killed? Are thread creations failing? Is performance suffering and paging to disk increasing? If the answer to these questions is no, then I don't think there's much to worry about.
Update based on some comments below and above:
First, 16KB is still pretty big for something that you say doesn't need much stack space. If you really want to go small, I would be tempted to say 4096 or 8192 on x86 Linux. Second, yes you can set your CPU's stack pointer to something else.. But when you malloc() or mmap(), that's going to take up space. I don't know how you think it's going to help to set the stack pointer to something else. That said, if you really feel strongly that the thread that calls main() has too big of a stack (I would say that is slightly crazy) and that pthread_attr_setstacksize() doesn't let you get small enough (?), then maybe you can look into non-portable stuff like creating threads by calling the clone() syscall and specifying stacks based on the main thread's stack pointer, or a buffer from elsewhere, or whatever. But you're still going to need a stack for each thread and I have a feeling top is still going to disappoint you. Maybe your expectations are a little high.
I have seen this problem as well. It is unclear how the stacks are accounted for but the "extra" space is counted against your total VM and if you run up against your process boundary you are in trouble (even though you aren't using the space). It seems dependent on what version of Linux you are running (even within the 2.6 family), and whether you are 32 bit or 64 bit.