I search for answer and so far haven't found a clear one.
I am doing testing which launches many threads calling "system()", like below.
for (int i = 0; i < 3000; ++i)
pthread_create(&thread[i], NULL, thread_func, NULL);
for (int i = 0; i < 3000; ++i)
pthread_join(thread[i], NULL);
...
void* thread_func(void* arg)
{
if (system('test.sh') == -1)
{
perror("system");
exit(1);
}
pthread_exit(NULL);
}
test.sh
#!/bin/bash
sleep 100
When I run the program, at certain point it will display.
system: Resource temporarily unavailable
Is there way to know which resource? I fix the max processes issue so I think it may be due to something else.
This error means that some system call called by the system library function returned EGAIN. Most likely is the fork call, which can fail with EAGAIN for a number of reasons:
EAGAIN A system-imposed limit on the number of threads was encountered. There are a num‐
ber of limits that may trigger this error:
* the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the
number of processes and threads for a real user ID, was reached;
* the kernel's system-wide limit on the number of processes and threads,
/proc/sys/kernel/threads-max, was reached (see proc(5));
* the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5));
or
* the PID limit (pids.max) imposed by the cgroup "process number" (PIDs) con‐
troller was reached.
Related
I'm trying to use RSS to estimate the mem usage of my application in linux.
for (int i = 0; i < 100; ++i) {
std::cout << "loading map " << i << std::endl;
{
process_mem_usage();
MyApplicaiton app();
// do things
process_mem_usage();
}
}
the process_mem_usage is basically monitoring the vm_size and rss using this approach How to get memory usage at runtime using C++?
By running this small bench, I only see RSS increase at the first time, and then keep the same.
I was able to claim that there is no mem leak(otherwise RSS should keep increasing). The only explanation is that the process didn't return the memory to the system (even if that memory is currently not used by my application). Is there any way to force the process return the memory to system? (I tried sleeping for a long time but didn't work).
Another example:
process_mem_usage();
{
std::shared_ptr<char> tmp((char *)operator new(500 * 1024 * 1024));
std::memset(tmp.get(), 1, 500 * 1024 * 1024);
process_mem_usage();
}
by running this example I can see RSS increase and then decrease immediately right after I destroy the shared_ptr.
So it's hard to explain what's going on under the hood.
My code is like below:
my $tcount :shared = 0;
while( $message = $msgQueue->poll()){
if($tcount < 4){
Logger::log(3, "Number of active threads : ".$tcount." processing incomming message");
threads->create(processData,$message,createMsgId());
} else {
Logger::log(2, "Maximum number of threads reached. Waiting");
sleep 1 until $tcount < 4);
threads->create(processData,$message,createMsgId());
}
}
sub processData {
$tcount++;
# do something
$tcount--;
threads->exit();
}
As per my understanding I am not creating more than 4 threads at a time. However, after running this script for a while I get:
Thread creation failed: pthread_create returned 11
What wrong am I doing?
When pthread_create returns EAGAIN, it means
Insufficient resources to create another thread, or a system-imposed limit on the number of threads was encountered. The latter case may occur in two ways: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of process for a real user ID, was reached; or the kernel's system-wide limit on the number of threads, /proc/sys/kernel/threads-max, was reached.
Well, at least on my system.
I'm trying to implement a syscall which allows me to get the number of threads for the current process. I am new to the Linux kernel, and so my understanding of it is limited.
Currently, I am trying to iterate through all the task_structs, and compare their thread group leader's PID with the current thread group leader's PID:
// ...
int nthreads = 0;
struct task_struct *task_it;
for_each_process(task_it) {
if (task_it->group_leader->pid == current->group_leader->pid) {
nthreads++;
}
}
// ...
However, this doesn't seem to be working (a quick test spawning some pthreads is still giving 1. What about the group_leader is common to all threads in the same process?
The problem with your code is that what the kernel calls a PID (the pid field of task_struct) is what userspace calls a TID (ie. it's what's returned by sys_gettid() and is unique per thread). What userspace calls a PID is called a TGID in the kernel (for "task group ID") - that's what the sys_getpid() syscall returns.
You don't need to actually check the TGID, though - just comparing the struct task_struct * pointers is enough:
if (task_it->group_leader == current->group_leader) {
By the way, you could just iterate over the thread_group list that current is a member of (with while_each_thread()), then you wouldn't need any test at all. Or even better, just use get_nr_threads(current).
Note that all methods that loop over the task lists need to be wrapped in rcu_read_lock(); / rcu_read_unlock(); to be correct.
This chunk of code is a good demonstration.
The following C program creates a list of all processes in the process
table of a node and shows in one column the number of threads for any
single process. Using this tool, it was possible to identify that the
network daemon created a new thread anytime a network problem
occurred. A severe network problem was responsible for the logon
problems.
#include "sys/param.h"
#include "sys/pstat.h"
int main ( void )
{
struct pst_status * psa = NULL;
struct pst_status * prc = NULL;
struct pst_dynamic psd;
long nproc = 0;
long thsum = 0;
long i;
if ( pstat_getdynamic(&psd, sizeof(psd), 1, 0) == -1 )
(void)perror("pstat_getdynamic failed");
// Get the number of active processes from pst_dynamic
nproc = psd.psd_activeprocs;
psa = (struct pst_status *)malloc(nproc * sizeof(struct pst_status));
// Read the info about the active processes into the array 'psa'
if ( pstat_getproc(psa, sizeof(struct pst_status), nproc, 0) == -1 )
(void)perror("pstat_getproc failed");
(void)printf("\n\n------------------------------------------------------------------------------");
(void)printf("\n %5s | %5s |%7s| %5s | %s", "PID", "UID", "Threads", "RSS", "Command");
(void)printf("\n------------------------------------------------------------------------------");
// Report the process info as required
prc = (struct pst_status *)psa;
for (i=0; i < nproc; i++)
{
(void)printf("\n %5ld | ", prc->pst_pid);
(void)printf("%5ld | ", prc->pst_uid);
(void)printf("%5ld | ", prc->pst_nlwps);
(void)printf("%5ld | ", prc->pst_rssize);
(void)printf("%s ", prc->pst_cmd);
thsum += prc->pst_nlwps;
++prc;
}
(void)printf("\n\n*** %ld processes, %ld threads running\n\n", nproc, thsum);
(void)free(psa);
(void)exit(0);
}
Found here:
http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801?ciid=060818f70fe0211018f70fe02110275d6e10RCRD
Here's another link using task_struct:
http://tuxthink.blogspot.com/2011/03/using-foreachprocess-in-proc-entry.html
I wrote a simple program to calculate the maximum number of threads that a process can have in linux (Centos 5). here is the code:
int main()
{
pthread_t thrd[400];
for(int i=0;i<400;i++)
{
int err=pthread_create(&thrd[i],NULL,thread,(void*)i);
if(err!=0)
cout << "thread creation failed: " << i <<" error code: " << err << endl;
}
return 0;
}
void * thread(void* i)
{
sleep(100);//make the thread still alive
return 0;
}
I figured out that max number for threads is only 300!? What if i need more than that?
I have to mention that pthread_create returns 12 as error code.
Thanks before
There is a thread limit for linux and it can be modified runtime by writing desired limit to /proc/sys/kernel/threads-max. The default value is computed from the available system memory. In addition to that limit, there's also another limit: /proc/sys/vm/max_map_count which limits the maximum mmapped segments and at least recent kernels will mmap memory per thread. It should be safe to increase that limit a lot if you hit it.
However, the limit you're hitting is lack of virtual memory in 32bit operating system. Install a 64 bit linux if your hardware supports it and you'll be fine. I can easily start 30000 threads with a stack size of 8MB. The system has a single Core 2 Duo + 8 GB of system memory (I'm using 5 GB for other stuff in the same time) and it's running 64 bit Ubuntu with kernel 2.6.32. Note that memory overcommit (/proc/sys/vm/overcommit_memory) must be allowed because otherwise system would need at least 240 GB of committable memory (sum of real memory and swap space).
If you need lots of threads and cannot use 64 bit system your only choice is to minimize the memory usage per thread to conserve virtual memory. Start with requesting as little stack as you can live with.
Your system limits may not be allowing you to map the stacks of all the threads you require. Look at /proc/sys/vm/max_map_count, and see this answer. I'm not 100% sure this is your problem, because most people run into problems at much larger thread counts.
I had also encountered the same problem when my number of threads crosses some threshold.
It was because of the user level limit (number of process a user can run at a time) set to 1024 in /etc/security/limits.conf .
so check your /etc/security/limits.conf and look for entry:-
username -/soft/hard -nproc 1024
change it to some larger values to something 100k(requires sudo privileges/root) and it should work for you.
To learn more about security policy ,see http://linux.die.net/man/5/limits.conf.
check the stack size per thread with ulimit, in my case Redhat Linux 2.6:
ulimit -a
...
stack size (kbytes, -s) 10240
Each of your threads will get this amount of memory (10MB) assigned for it's stack. With a 32bit program and a maximum address space of 4GB, that is a maximum of only 4096MB / 10MB = 409 threads !!! Minus program code, minus heap-space will probably lead to your observed max. of 300 threads.
You should be able to raise this by compiling a 64bit application or setting ulimit -s 8192 or even ulimit -s 4096. But if this is advisable is another discussion...
You will run out of memory too unless u shrink the default thread stack size. Its 10MB on our version of linux.
EDIT:
Error code 12 = out of memory, so I think the 1mb stack is still too big for you. Compiled for 32 bit, I can get a 100k stack to give me 30k threads. Beyond 30k threads I get Error code 11 which means no more threads allowed. A 1MB stack gives me about 4k threads before error code 12. 10MB gives me 427 threads. 100MB gives me 42 threads. 1 GB gives me 4... We have 64 bit OS with 64 GB ram. Is your OS 32 bit? When I compile for 64bit, I can use any stack size I want and get the limit of threads.
Also I noticed if i turn the profiling stuff (Tools|Profiling) on for netbeans and run from the ide...I only can get 400 threads. Weird. Netbeans also dies if you use up all the threads.
Here is a test app you can run:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <signal.h>
// this prevents the compiler from reordering code over this COMPILER_BARRIER
// this doesnt do anything
#define COMPILER_BARRIER() __asm__ __volatile__ ("" ::: "memory")
sigset_t _fSigSet;
volatile int _cActive = 0;
pthread_t thrd[1000000];
void * thread(void *i)
{
int nSig, cActive;
cActive = __sync_fetch_and_add(&_cActive, 1);
COMPILER_BARRIER(); // make sure the active count is incremented before sigwait
// sigwait is a handy way to sleep a thread and wake it on command
sigwait(&_fSigSet, &nSig); //make the thread still alive
COMPILER_BARRIER(); // make sure the active count is decrimented after sigwait
cActive = __sync_fetch_and_add(&_cActive, -1);
//printf("%d(%d) ", i, cActive);
return 0;
}
int main(int argc, char** argv)
{
pthread_attr_t attr;
int cThreadRequest, cThreads, i, err, cActive, cbStack;
cbStack = (argc > 1) ? atoi(argv[1]) : 0x100000;
cThreadRequest = (argc > 2) ? atoi(argv[2]) : 30000;
sigemptyset(&_fSigSet);
sigaddset(&_fSigSet, SIGUSR1);
sigaddset(&_fSigSet, SIGSEGV);
printf("Start\n");
pthread_attr_init(&attr);
if ((err = pthread_attr_setstacksize(&attr, cbStack)) != 0)
printf("pthread_attr_setstacksize failed: err: %d %s\n", err, strerror(err));
for (i = 0; i < cThreadRequest; i++)
{
if ((err = pthread_create(&thrd[i], &attr, thread, (void*)i)) != 0)
{
printf("pthread_create failed on thread %d, error code: %d %s\n",
i, err, strerror(err));
break;
}
}
cThreads = i;
printf("\n");
// wait for threads to all be created, although we might not wait for
// all threads to make it through sigwait
while (1)
{
cActive = _cActive;
if (cActive == cThreads)
break;
printf("Waiting A %d/%d,", cActive, cThreads);
sched_yield();
}
// wake em all up so they exit
for (i = 0; i < cThreads; i++)
pthread_kill(thrd[i], SIGUSR1);
// wait for them all to exit, although we might be able to exit before
// the last thread returns
while (1)
{
cActive = _cActive;
if (!cActive)
break;
printf("Waiting B %d/%d,", cActive, cThreads);
sched_yield();
}
printf("\nDone. Threads requested: %d. Threads created: %d. StackSize=%lfmb\n",
cThreadRequest, cThreads, (double)cbStack/0x100000);
return 0;
}
At some point during my C programming adventures on Linux, I encountered flags (possibly ioctl/fcntl?), that make reads and writes on a file descriptor uninterruptible.
Unfortunately I cannot recall how to do this, or where I read it. Can anyone shed some light?
Update0
To refine my query, I'm after the same blocking and guarantees that fwrite() and fread() provide, sans userspace buffering.
You can avoid EINTR from read() and write() by ensuring all your signal handlers are installed with the SA_RESTART flag of sigaction().
However this does not protect you from short reads / writes. This is only possible by putting the read() / write() into a loop (it does not require an additional buffer beyond the one that must already be supplied to the read() / write() call.)
Such a loop would look like:
/* If return value is less than `count', then errno == 0 indicates end of file,
* otherwise errno indicates the error that occurred. */
ssize_t hard_read(int fd, void *buf, size_t count)
{
ssize_t rv;
ssize_t total_read = 0;
while (total_read < count)
{
rv = read(fd, (char *)buf + total_read, count - total_read);
if (rv == 0)
errno = 0;
if (rv < 1)
if (errno == EINTR)
continue;
else
break;
total_read += rv;
}
return rv;
}
Do you wish to disable interrupts while reading/writing, or guarantee that nobody else will read/write the file while you are?
For the second, you can use fcntl()'s F_GETLK, F_SETLK and F_SETLKW to acquire, release and test for record locks respectively. However, since POSIX locks are only advisory, Linux does not enforce them - it's only meaningful between cooperating processes.
The first task involves diving into ring zero and disabling interrupts on your local processor (or all, if you're on an SMP system). Remember to enable them again when you're done!