We are seeing a big amount of ptypes posix threads hanging in socket poll even though the call is made with a timeout of 200 ms. Any ideas what can lead to such a situation?
Related
I'm blocking certain signals in my process, and I'm wondering what is going to happen if the pending signals limit is exceeded.
Do the new ones get lost or is my process going to crash?
when i using go, i got a error 'runtime: program exceeds 10000-thread limit'.
That is the information about goroutine.
'SCHED 229357ms: gomaxprocs=16 idleprocs=0 threads=8797 idlethreads=8374 runqueue=2131 gcwaiting=0 nmidlelocked=141 nmspinning=0 stopwait=0 sysmonwait=0'
we can see that has 8374 idlethreads, no reason to create any more thread by os.
why the program exceeds 10000-thread limit?
No way to provide a definitive answer without a lot of program code. And I doubt that you can provide a simplified example that will hit the problem. 10,000 threads is a lot of threads.
What I guess is that because Go creates a new thread for each goroutine's blocking operations, you have a lot of goroutines doing blocking calls.
I am not sure, but I think each goroutine keeps its own blocking call thread and they are not pooled. So having more than 10,000 goroutines that each make blocking calls may be the issue.
All guesses though.
EDIT:
Found a way to increase the 10,000 thread limit:
https://golang.org/pkg/runtime/debug/#SetMaxThreads
debug.SetMaxThreads(20000)
I've been searching and reading about killing threads (C posix threads), and everybody says that is not a good idea because a thread should make its work and then return... but my problem is the next:
I'm reciving messages in my local network (using the recvfrom function), but this function "blocks" my program, I mean, if I don't revice any messege the function keeps locked (forever) until it recives something.
To avoid this, I thought to use threads, so, while my main thread is "counting", my second thread is try to recive messages. If in a determinated time (i.e. 1 second), my second thread is still waiting for a message (is locked in the recvfrom function) I need to "kill it" and then create another thread to start again (and try to recive messages from another IP). This means that not always my thread going to finish its work and I can't wait forever...
So far I can do that (create a lot of threads and recive the messages from the IP I'm interested in), but I don't know how to kill the threads that never recived anything...
Someone knows how to kill the threads? Or they are killed automatically when my main program returns?
Thank you and really sorry for my poor english...
Looks like its related to one of my questions How to avoid thread waiting in the following or similar scenarios (want to make a thread wait iff its really really necessary)?
But its .net, though (code sample is in C#)
Essentially i spawned new thread and performing some i/o oeprations and its a blocking call.
And for some reason it just waits foreve, i do have timeout so that i can abort the thread 'abort' method.
Rearchitect so the thread can receive messages from any IP. That way, you can try to receive messages from another IP without having to disturb the thread.
What causes these sleeping processes that I see in top? If I were to call PHP's sleep() function, would that add to the sleeping count I see in top? Are there any disadvantages to having a high number in sleeping?
A process is sleeping when it is blocked, waiting for something. For example, it might have called read() and is waiting on data to arrive from a network stream.
sleep() is indeed one way to have your process sleep for a while. Sleeping is, however, the normal state of all but heavily compute-bound processes - sleeping is essentially what a process does when it isn't doing anything else. It's the normal state of affairs for most of your processes to be sleeping - if that's not the case, it tends to indicate that you need more CPU horsepower.
A sleeping process is like suspended process.
A process sleeps when:
It's doing an I/O operation (blocking for I/O)
When you order it to sleep by sleep()
The status of any process can be:
Ready: when it ready for execution and it's in the queue waiting the processor call with specific priority
Sleeping: When it was running and it was blocked for I/O operation or when executing sleep()
Running: When the processor executes a process it becomes running.
Status Meaning
R Runnable
T Stopped
P Waiting on Pagein
D Waiting on I/O
S Sleeping < 20 seconds
I Idle - sleeping >20 seconds
Z Zombie or defunct
They are processes which aren't running on the CPU right now. This is not necessarily a bad thing.
If you have huge numbers (10,000 on a server system, for example) of processes sleeping, the amount of memory etc used to keep track of them may make the system less efficient for non-sleeping processes.
Otherwise, it's fine.
Most normal server systems have 100 to 1000 much of the time; this is not a big deal.
Just because they're not doing anything just now doesn't mean they won't, very soon. Keeping them in memory, ready, reduces latency when they are required.
To go into a bit more detail here, the S state means the process is waiting on a timer or a slow device, while the D state means it is waiting on a fast device.
What constitutes a fast device vs a slow device is not terribly well defined, but generally, all serial, network, and terminal devices are slow devices, while disks are fast devices.
I am writing a server application in which there is a thread deployed to read/write many sockets connecting to clients. My manager tells me that it is not a good design, because if the thread aborts due to unknown reason then all the read/write work will stop forever.
So I wonder in what conditions will a thread abort, except the case we return from the Run() function of a thread. Do we need consider the case that the thread stops running abnormally?
It depends. One thread per client can be a bad thing scalability wise, especially if the thread doesn't do that much work per client. In that circumstance it can be better to have a thread that handles a number of clients, the idea to achieve a good balance between the number of threads and having them do a decent amount of work.
If on the other hand each thread is doing a lot of work per client then one thread isn't such a bad idea, the overhead of the thread not being significant in comparison to the work load.
So setting that aside, a thread will abort if your code is written so that the thread returns or self-terminates. If another thread in your program knows the thread's handle/id then the library you're using may have a function with a name like thread_kill(). That would allow that other thread to kill this thread, though that's almost always a bad idea.
So as far as I'm concerned your thread will only abort and disappear if you've written your code to make that happen deliberately.
Handling exceptions is probably best done in its entirety within the thread where the exception arose. I've never tried to do otherwise (still writing in pure C), but the word is that it's difficult to handle them outside the thread. Irrespective of whether each thread handles one or many clients you still have to handle all errors and events within thread.
It may be simpler to get that correct if you write I so that a thread handles handles a single client. Getting it wrong could lead to a thread getting into a stalled state (eg waiting for the client that is listening too) and accumulating those as time goes past will eventually kill your whole system.
I am writing a server application in which there is a thread deployed to read/write many sockets connecting to clients.
Not a good design. There should be at least one thread per client, in some circumstances two: one to read and one to write. If you're dealing in blocking I/O, servicing one client could block out all the others. (If you're dealing in non-blocking I/O you don't need threads at all.)
My manager tells me that it is not a good design, because if the thread aborts due to unknown reason then all the read/write work will stop forever.
He's right, for more reasons than he is advancing.