Have 2 threads, T1 is getting stuck executing a call.
And at the same time another thread T2 gets disconnected.
During T2's disconnection it performs clean up of T1 and then itself.
Since T1 is blocked. T2 is also not responding and its causing other unwanted behavior of the software.
One possible soln is to have a wait limit on T1. Is this the best way to handle?
A solution is to introduce a third thread:
T3 creates T1 and T2. T1 is executing a call and is blocked. T2 is finished, but does not need to wait for T1, because the clean up is the responsibility of T3. There is no unwanted behaviour of T2 being blocked.
Related
I'm very new to Dart and stilling learning it. As I understand, Dart executes code in different isolates. An isolate could start up another isolate to execute some long-running code. For each isolate, there is a thread and some memory allocated for it. These isolates are isolated like a bunch of little VMs.
I also read from the Dart document that Dart is a single threaded language. But, think about it, each isolate has its own thread. If isolate A has thread t1 and isolate B has thread t2, t1 and t2 are not the same thread, right?
If t1 and t2 are the same thread, then t1 and t2 can't execute code at the same time, which is ridiculous. So, t1 and t2 must be different thread.
If so, why we say Dart is a single-threaded language?
Yes and no.
"Yes" in the sense that you don't have to worry about locks or mutexes.
"No" in the sense that you list.
Dart tries to offer some of the benefits of multi-threading with isolates while avoiding all of the issues with shared memory multi-threading.
I am working on an application that has numerous threads with each thread performing a task that is different from the other. There is also a healthMonitor thread that monitors health of individual threads by sending heartbeat messages at regular intervals. Among these threads there is one thread T1 that posts messages to another thread T2 in response to which T2 does some file write operation.
On rare occasions, T1 sends a surge of messages for a prolonged duration to T2, because of which T2 becomes busy with file write operations. Sometimes these file-write operations can be time consuming taking up to 2 seconds to complete. In this situation, T2 does not respond to the heartbeat messages posted by the healthMonitor thread, causing the healthMonitor thread to kill the application.
T2 reads all incoming messages including those posted by T1 and the hearbeat messages posted by the healthMonitor thread, from the same messageQueue.
This issue occurs very rarely and cannot be re-produced at will thus making it difficult to debug.
I have concluded that during such situations where there are huge number of messages pending to be processed and with the processing time taking too long, the heartbeat messages remain unprocessed for too long. Because of this thread T2 does not respond to the healthMonitor thread on time.
My question is, what would be the proper way to handle this issue. I have few options in mind:
Limit the number of messages from thread T1. This approach would reduce the load from thread T2, but is this an elegant way to handle the situation?
Implement priority-based handling in the message queue: All heartbeat messages can be added to the head of the message queue instead of getting added to the tail so that it gets picked up first from the queue. Please note that we have our own implementation of message queue for inter-thread communication.
Or should I just focus on optimizing the file write operation in thread T2?
I have 3 threads in my program
t1
reads a frame1 of data and
writes it onto a hard disk
t2
reads a frame2 of data and
writes it onto a hard disk
t2
reads a frame3 of data and
writes it onto a hard disk
When the program runs, and t1 t2 and t3 are scheduled for execution one by one, how are the operations performed internally?
Ex:
say t1 -> t2 -> t3 get scheduled in this order
Scenario 1:
will t1 finish one full cycle of read frame1 and write frame1 before t2 is scheduled and whether t2 finishes one full cycle of read frame2 and write frame2 before t3 is scheduled and so on?
or
Scenario 2:
can t1 or t2 or t3 or few or all of these threads' execution be stopped in the middle of their execution before the next thread gets scheduled?
Which of these scenarios is correct?
I am especially mentioning hard disk write as there is a possibility of a blocking fwrite call, which cannot be left in the middle of its execution
You should consider (and code and think) as if all threads are running concurrently (e.g. a the same time on different cores of your processor).
A thread usually don't write directly to the disk: it is writing files to some file system (and the kernel is buffering, e.g. in the page cache, so the disk IO can happen several seconds later).
If you need synchronization, you should make it explicitly (e.g. with mutexes). If you need to synchronize file contents, consider using some file locking machinery à la lockf(3) (but you should really avoid having several threads or processes accessing and writing the same file). BTW stdio is buffered (so you might want to fflush(3) after fwrite(3)...)
And when the kernel is scheduling some thread or process, it will schedule preemptively at arbitrary time (at any machine instruction).
Read some pthread tutorial and Operating Systems: Three Easy Pieces. Read also about memory models (it is tricky).
So all your scenarii could and are likely to be wrong.
How much of a thread's code get executed ever time it is scheduled?
You should not care, and you cannot know. It can be as tiny as nothing (read about thrashing), and as large as several millions machine instructions. BTW, be aware of optimizing compilers and of sequence points in C; so actually the question does not even make sense (from the observable point of view of a C programmer).
I am especially mentioning hard disk write as there is a possibility of a blocking fwrite call
When the stdio library (or directly your application) is actually write(2)-ing a file descriptor, it is likely (but not certain) that the kernel will schedule tasks during such system calls. However, the actual disk IO will happen probably later.
PS. Read also about undefined behavior.
It depends on the method (or methods) these threads are calling. If all these threads are calling a same method and if that method is synchronized then only one thread will be processing it at a time. During that time rest of the threads will wait for currently running thread to complete. If not synchronized or threads are calling different methods then there is no guarantee which thread will get processed first or finish first. They also may end up overwriting class-level variables.
Suppose there are 4 Threads (T1 to T4) that need to run concurrently and 3 structs (Struct1 to Struct3) as resources
T1 to T2 share struct1 (by T1 writing to struct1 and T2 reading from it)
T2 to T3 share struct2 (by T2 writing to struct2 and T3 reading from it)
T3 to T4 share struct3 (by T3 writing to struct3 and T4 reading from it)
Because of this statement from § 41.2.4 of The C++ Programming Language (4th edition) by Bjarne Stroustrup :
"Two threads have a data race if both can access a memory location
simultaneously and at least one of their accesses is a write. Note
that defining “simultaneously” precisely is not trivial. If two
threads have a data race, no language guarantees hold: the behavior is
undefined."
It becomes clear there is a need for syncrhonization.
1 - Which of these primitives are suitable to this application , just mutices or Semaphores ?
2- If mutex is the choice, we would need 3 mutices, one mutex for each structure , right ?
3- Would the fact of using a mutex at a given non-atomic operation, block CPU time of other threads ?
Your usecase is kind of abstract so batter solutions might be available. But based just on the information you provided:
1) Use mutex. I do not see how semaphores could help except to be used as mutex. A semaphore could be usefull when you share more resources, but in your case it is only one at a time.
If all four threads would access the first free struct or if your struct would be an queue, a semaphore could help.
2) Right, one mutex per structure.
3) Yes, it could, this is the idea, you do not want for T1 to write when T2 is reading struct1 and viceversa. Worstcase could be T1 blocks T2 that has already blocked T3 that has blocked T4.
1 - 3 semaphore for each queue, see Producer–consumer problem.
2- 1 of the semaphores could be a mutex, binary semaphores are much like mutex.
3- if you have to wait for a semaphore or mutex you will be placed in the no ready queue of the OS, waiting for the release. And so doesn't use any CPU (except for the 1000's of cycles it cost for the context switch).
This is an interview question.
How to detect and find out if a program is in deadlock? Are there some tools that can be used to do that on Linux/Unix systems?
My idea:
If a program makes no progress and its status is running, it is deadlock. But, other reasons can also cause this problem. Open source tools are valgrind (halgrind) can do that. Right?
If you suspect a deadlock, do a ps aux | grep <exe name>, if in output, the PROCESS STATE CODE is D (Uninterruptible sleep) means it is a deadlock.
Because as #daijo explained, say you have two threads T1 & T2 and two critical sections each protected by semaphores S1 & S2 then if T1 acquires S1 and T2 acquires S2 and after that they try to acquire the other lock before relinquishing the one already held by them, this will lead to a deadlock and on doing a ps aux | grep <exe name>, the process state code will be D (ie Uninterruptible sleep).
Tools:
Valgrind, Lockdep (linux kernel utility)
Check this link on types of deadlocks and how to avoid them :
http://cmdlinelinux.blogspot.com/2014/01/linux-kernel-deadlocks-and-how-to-avoid.html
Edit: ps aux output D "could" mean process is in deadlock, from this redhat doc:
Uninterruptible Sleep State
An Uninterruptible sleep state is one
that won't handle a signal right away. It will wake only as a result
of a waited-upon resource becoming available or after a time-out
occurs during that wait (if the time-out is specified when the process
is put to sleep).
I would suggest you look at Helgrind: a thread error detector.
The simplest example of such a problem is as follows.
Imagine some shared resource R, which, for whatever reason, is guarded by two locks, L1 and L2, which must both be held when R is accessed.
Suppose a thread acquires L1, then L2, and proceeds to access R. The implication of this is that all threads in the program must acquire the two locks in the order first L1 then L2. Not doing so risks deadlock.
The deadlock could happen if two threads -- call them T1 and T2 -- both want to access R. Suppose T1 acquires L1 first, and T2 acquires L2 first. Then T1 tries to acquire L2, and T2 tries to acquire L1, but those locks are both already held. So T1 and T2 become deadlocked."