.Net Critical regions in threading not working as desired - multithreading

I am trying to run a sample code (a very basic one) involving threading and critical regions.
Here's my code:
public static void DoCriticalWork(object o)
{
SomeClass instance = o as SomeClass;
Thread.BeginCriticalRegion();
instance.IsValid = true;
Thread.Sleep(2);
instance.IsComplete = true;
Thread.EndCriticalRegion();
instance.Print();
}
And I am calling it as follows:
private static void CriticalHandled()
{
SomeClass instance = new SomeClass();
ParameterizedThreadStart operation = new ParameterizedThreadStart(CriticalRegion.DoCriticalWork);
Thread t = new Thread(operation);
Console.WriteLine("Start thread");
t.Start(instance);
Thread.Sleep(1);
Console.WriteLine("Abort thread");
t.Abort();
Console.WriteLine("In main");
instance.Print();
}
However, the output I get is:
**
Start thread
Abort thread
In main
IsValid: True
IsComplete: False
**
Since the critical region is defined, IsComplete should be true and not false.
Can someone please explain why it is not working?
Here is SomeClass for reference:
public class SomeClass
{
private bool _isValid;
public bool IsValid
{
get { return _isValid; }
set { _isValid = value; }
}
private bool _isComplete;
public bool IsComplete
{
get { return _isComplete; }
set { _isComplete = value; }
}
public void Print()
{
Console.WriteLine("IsValid: {0}", IsValid);
Console.WriteLine("IsComplete: {0}", IsComplete);
Console.WriteLine();
}
}
Edit
Expln from MCTS notes:
The idea behind a critical region is to provide a region of code that must be executed as if it were a single line. Any attempt to abort a thread while it is within a critical region will have to wait until after the critical region is complete. At that point, the thread will be aborted, throwing the ThreadAbortException. The difference between a thread with and without a critical region is illustrated in the following figure:
Critical Region http://www.freeimagehosting.net/uploads/9dd3bb5445.gif

Thread.BeginCriticalRegion does not prevent a Thread from being aborted. I believe it is used to notify the runtime that if the Thread is aborted in the critical section, it is not necessarily safe to continue running the application/AppDomain.
The MSDN docs have a more complete explanation: http://msdn.microsoft.com/en-us/library/system.threading.thread.begincriticalregion.aspx

Its a sleep timing problem. Just widen the gap between the two sleeps and you will get the answer.
There are two threads: The main thread and the thread that does the critical work. Now when abort is called the thread 't' will abort instantly even if it has not completed the critical region.
Now as you have send the main thread to sleep for 2ms and thread t for 1ms, sometimes t will complete the critical section and sometimes it will not. So that's why the value for IsComplete is sometimes false and sometimes true.
Now just send the main thread to sleep for 100ms and you will find the IsComplete is always true.
Vice versa send the thread "t" to sleep for 100ms and you will find the IsComplete is always false.
EDIT
FROM MSDN
Notifies a host that execution is about to enter a region of code in which the effects of a thread abort or unhandled exception might jeopardize other tasks in the application domain.
For example, consider a task that attempts to allocate memory while holding a lock. If the memory allocation fails, aborting the current task is not sufficient to ensure stability of the AppDomain, because there can be other tasks in the domain waiting for the same lock. If the current task is terminated, other tasks could be deadlocked.
When a failure occurs in a critical region, the host might decide to unload the entire AppDomain rather than take the risk of continuing execution in a potentially unstable state. To inform the host that your code is entering a critical region, call BeginCriticalRegion. Call EndCriticalRegion when execution returns to a non-critical region of code.
From CLR Inside Out: Writing Reliable Code
State Corruption
There are three buckets that state corruption may fall into. The first is local state, which includes local variables and heap objects that are only used by a particular thread. The second is shared state, which includes anything shared across threads within the AppDomain, such as objects stored in static variables. Caches often fall into this category. The third is process-wide, machine-wide, and cross-machine shared state—files, sockets, shared memory, and distributed lock managers fall into this camp.
The amount of state that can be corrupted by an async exception is the maximum amount of state a thread is currently modifying. If a thread allocates a few temporary objects and doesn't expose them to other threads, only those temporary objects can be corrupted. But if a thread is writing to shared state, that shared resource may be corrupted, and other threads may potentially encounter this corrupted state. You must not let that happen. In this case, you abort all the other threads in the AppDomain and then unload the AppDomain. In this way, an asynchronous exception escalates to an AppDomain, causing it to unload and ensuring that any potentially corrupted state is thrown away. Given a transacted store like a database, this AppDomain recycling provides resiliency to corruption of local and shared state.
Critical Regions allow you to handle situations where some piece of code might corrupt other Application Domains and causing irreparable damage to the system.

A good solution is to encapsulate the code from DoCriticalWork() with
try { ... } catch(ThreadAbortedException) {...}
where you should act as you wish (maybe set IsComplete = true?)
You can read more about ThreadAbortException, and be sure to check Thread.ResetAbort and the finally block usage for this case.
As Andy mentioned and quoting from Thread.BeginCriticalRegion:
Notifies a host that execution is about to enter a region of code in which the effects of a thread abort or unhandled exception might jeopardize other tasks in the application domain.

Related

Calling the instance to the thread inside that same thread

Context:
I have a cmd application in java which is written to work in peer-to-peer mode in different servers. Once a server starts, all other instances must stop. So I have written a piece of code that runs in a low priority thread and monitors an AtomicBoolean value autoClose, and whenever autoClose is set to true, thread will close application. (P.S.: I don't want to manually add close because the application has 2 main high priority threads and many temporary normal priority threads).
Here is the code:
/**
* Watches autoClose boolean value and closes the connector once it is true
* <p>
* This is a very low priority thread which continuously monitors autoClose
*/
protected void watchAndClose() {
Thread watchAutoClose = new Thread(() -> {
while (true) {
if (autoClose.get()) {
close();
// wait till closing is successful
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException ignored) {
// I want instance of thread watchAutoClose so I can call this
// watchAutoClose.interrupt();
}
if (!component.getStatus()) setAutoClose(false);
}
}
});
watchAutoClose.setPriority(Thread.MIN_PRIORITY);
watchAutoClose.start();
}
Question:
SonarLint says I can't leave InterruptedException part empty. I have to either throw it again or call thatThread.interrupt().
So how can I do this? I want an instance of thread watchAutoClose inside that thread so I can call watchAutoClose.interrupt(). I tried Thread.currentThread() but I fear with that many threads, the currently executing thread wouldn't be this thread. (i.e, there is a possibility of JVM can choose to switch to another thread by the time it is inside the catch clause and calls Thread.currentThread() so at that time current thread would be the other one and I would interrupt that other thread... correct me if I am too worrying or my concept is totally wrong.)
Or should I ignore the warning altogether and leave catch block?
First of all, it’s not clear why you think that waiting for a second was necessary at all. By the time, the close() method returns, the close() method has been completed. On the other hand, if close() truly triggers some asynchronous action, there is no guaranty that waiting one second will be sufficient for its completion.
Further, addressing your literal question, Thread.currentThread() always return the calling thread’s instance. It’s impossible for a thread to execute that method without being in the running state. When a task switch happens, the thread can’t read the reference at all, until it gets CPU time again. Besides that, since the specification says that this method returns the Thread instance representing the caller, the environment has to ensure this property, regardless of how it implements it. It works even when multiple threads call this method truly at the same time, on different CPU cores.
So, regardless of how questionable the approach of waiting a second is, handling interruption like
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
}
is a valid approach.
But you may also replace this code with
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(1));
The parkNanos method will return silently on interruption, leaving the calling thread in the interrupted state. So it has the same effect as catching the InterruptedException and restoring the interrupted state, but is simpler and potentially more efficient as no exception needs to be constructed, thrown, and caught.
Another point is that you are creating a polling loop on the atomic variable consuming CPU cycles when the variable is false, which is discouraged, even when you give the thread a low priority.

Spawning a new thread for each object load

I have a system which runs multiple service (long lived) and worker (short lived) threads. They all share a state which contains objects. Any thread can request an object an any time, through a singleton-of-sorts class called ObjectManager. If the object is not available it needs to be loaded.
Here's some pseudo-code of how object loading looks now:
class ObjectManager {
getLoadinData(path) {
if (hasLoadingDataFor(path))
return whatWeHave()
else {
loadingData = createNewLoadingData();
loadingData.path = path;
pushLoadingTaskToLoadingThread(loadingData);
return loadingData;
}
}
// loads object and blocks until it's loaded
loadObjectSync(path) {
loadingData = getLoadinData(path);
waitFor(loadingData.conditionVar);
return loadingData.loadedObject;
}
// initiates a load and calls a callback when done
loadObjectAsync(path, callback) {
loadingData = getLoadinData(path);
loadingData.callbacks.add(callback);
}
// dedicated loading thread
loadingThread() {
while (running) {
loadingData = waitForLoadingData();
object = readObjectFromDisk(loadingData.path);
object.onLoaded(); // !!!!
loadingData.object = object;
// unblock cv waiters
loadingData.conditionVar.notifyAll();
// call callbacks
loadingData.callbacks.callAll(object);
}
}
}
The problem is the line object.onLoaded. I have no control over this function. Some objects might decide that they need other objects to be valid. So in their onLoaded method they might call loadObjectSync. Uh-oh! This (naturally) dead locks. It blocks the loading loop until the loading loop makes more iterations.
What I could do to solve this is leave the onLoaded call to the initiating threads. This will change loadObjectSync to something like:
loadObjectSync(path) {
loadingData = getLoadinData(path);
waitFor(loadingData.conditionVar);
if (loadingData.wasCreatedInThisThread()) {
object.onLoaded();
loadingData.onLoadedConditionVar.notifyAll();
loadingData.callbacks.callAll(object);
}
else {
// wait more
waitFor(loadingData.onLoadedConditionVar);
}
return loadingData.loadedObject;
}
... but then the problem is that if I have no calls for loadSync and only for loadAsync or simply the loadAsync call was the first to create the loading data, there will be no one to finalize the object. So to make this work, I have to introduce another thread finalizes objects whose loadingData was created by loadObjectAsync.
It seems that it would work. But I have a simpler idea! What if I change getLoadingData instead. What if it does this:
getLoadinData(path) {
if (hasLoadingDataFor(path))
return whatWeHave()
else {
loadingData = createNewLoadingData();
loadingData.path = path;
///
thread = spawnLoadingThread(loadingData);
thread.detach();
///
return loadingData;
}
}
Spawn a new thread for every object load. Thus there is no dead lock. Every loading thread can safely block until it's done. The rest of the code remains exactly as it is.
This means potentially tens (or why not thousands in certain edge cases) active threads, waiting on condition variables. I know that spawning threads has its overhead but I think it would be negligible compared to the cost of I/O from readObjectFromDisk
So my question is: Is this terrible? Can this somehow backfire?
The target platform is conventional desktop machines. But this software is supposed to run for a long time without stopping: weeks, maybe months.
Alternatively... even though I have an idea how to solve this if the thread-per-load turns out to be terrible, can this be solved in another way?
Very interesting! This is a problem I have bumped into a couple of times, trying to add a synchronous interface to a fundamentally asynchronous operation (i.e. file load, or in my case, network write) that is performed by a service thread.
My own preference would be to not provide the synchronous interface. Why? Because it keeps the code simpler in design & implementation and easier to reason about -- always important for multi-threading.
Benefits of sticking to single thread & async only is that you only have 1 service thread, so resource growth is not a concern, plus the user callbacks are always invoked on this same thread, which simplifies thread-safety concerns for users of ObjectManager (if you have multiple callback threads, every user callback must be thread safe, so it's an important choice to make). However sticking to only an async interface does mean the user of ObjectManager has more work to do.
But if you do want to keep the synchronous interface, then another approach that I have taken could work for you. You stick to a single service thread but inside the implementation of loadObjectSync you check the thread-ID to determine if the invoker is the service thread or any-other thread. If it is any-other thread you queue the request and safely block. But if it is the service thread, you can immediately load the object, say by calling a new function loadObjectImpl. You will need to grab the thread-ID of the service thread during initialization and store it within the ObjectManager instance, and use that for thread identification. And you will need a new function which is basically just the internal scope of the loadingThread function -- i.e. a new function called something like loadObjectImpl.

why does std::condition_variable::wait need mutex?

TL;DR
Why does std::condition_variable::wait needs a mutex as one of its variables?
Answer 1
You may look a the documentation and quote that:
wait... Atomically releases lock
But that's not a real reason. That's just validate my question even more: why does it need it in the first place?
Answer 2
predicate is most likely query the state of a shared resource and it must be lock guarded.
OK. fair.
Two questions here
Is it always true that predicate query the state of a shared resource? I assume yes. I t doesn't make sense to me to implement it otherwise
What if I do not pass any predicate (it is optional)?
Using predicate - lock makes sense
int i = 0;
void waits()
{
std::unique_lock<std::mutex> lk(cv_m);
cv.wait(lk, []{return i == 1;});
std::cout << i;
}
Not Using predicate - why can't we lock after the wait?
int i = 0;
void waits()
{
cv.wait(lk);
std::unique_lock<std::mutex> lk(cv_m);
std::cout << i;
}
Notes
I know that there are no harmful implications to this practice. I just don't know how to explain to my self why it was design this way?
Question
If predicate is optional and is not passed to wait, why do we need the lock?
When using a condition variable to wait for a condition, a thread performs the following sequence of steps:
It determines that the condition is not currently true.
It starts waiting for some other thread to make the condition true. This is the wait call.
For example, the condition might be that a queue has elements in it, and a thread might see that the queue is empty and wait for another thread to put things in the queue.
If another thread were to intercede between these two steps, it could make the condition true and notify on the condition variable before the first thread actually starts waiting. In this case, the waiting thread would not receive the notification, and it might never stop waiting.
The purpose of requiring the lock to be held is to prevent other threads from interceding like this. Additionally, the lock must be unlocked to allow other threads to do whatever we're waiting for, but it can't happen before the wait call because of the notify-before-wait problem, and it can't happen after the wait call because we can't do anything while we're waiting. It has to be part of the wait call, so wait has to know about the lock.
Now, you might look at the notify_* methods and notice that those methods don't require the lock to be held, so there's nothing actually stopping another thread from notifying between steps 1 and 2. However, a thread calling notify_* is supposed to hold the lock while performing whatever action it does to make the condition true, which is usually enough protection.
TL;DR
If predicate is optional and is not passed to wait, why do we need the lock?
condition_variable is designed to wait for a certain condition to come true, not to wait just for a notification. So to "catch" the "moment" when the condition becomes true you need to check the condition and wait for the notification. And to avoid a race condition you need those two to be a single atomic operation.
Purpose Of condition_variable:
Enable a program to implement this: do some action when a condition C holds.
Intended Protocol:
Condition producer changes state of the world from !C to C.
Condition consumer waits for C to happen and takes the action while/after condition C holds.
Simplification:
For simplicity (to limit number of cases to think of) let's assume that C never switches back to !C. Let's also forget about spurious wakeups. Even with this assumptions we'll see that the lock is necessary.
Naive Approach:
Let's have two threads with an essential code summarized like this:
void producer() {
_condition = true;
_condition_variable.notify_all();
}
void consumer() {
if (!_condition) {
_condition_variable.wait();
}
action();
}
The Problem:
The problem here is a race condition. A problematic interleaving of the threads is following:
The consumer reads condition, checks it to be false and decides to wait.
A thread scheduler interrupts consumer and resumes producer.
The producer updates condition to become true and invokes notify_all().
The consumer is resumed.
The consumer actually does wait(), but is never notified and waken up (a liveness hazard).
So without locking the consumer may miss the event of the condition becoming true.
Solution:
Disclaimer: this code still does not handle spurious wakeups and possibility of condition becoming false again.
void producer() {
{ std::unique_lock<std::mutex> l(_mutex);
_condition = true;
}
_condition_variable.notify_all();
}
void consumer() {
{ std::unique_lock<std::mutex> l(_mutex);
if (!_condition) {
_condition_variable.wait(l);
}
}
action();
}
Here we check condition, release lock and start waiting as a single atomic operation, preventing the race condition mentioned before.
See Also
Why Lock condition await must hold the lock
You need a std::unique_lock when using std::condition_variable for the same reason you need a std::FILE* when using std::fwrite and for the same reason a BasicLockable is necessary when using std::unique_lock itself.
The feature std::fwrite gives you, entire the reason it exists, is to write to files. So you have to give it a file. The feature std::unique_lock provides you is RAII locking and unlocking of a mutex (or another BasicLockable, like std::shared_mutex, etc.) so you have to give it something to lock and unlock.
The feature std::condition_variable provides, the entire reason it exists, is the atomically waiting and unlocking a lock (and completing a wait and locking). So you have to give it something to lock.
Why would someone want that is a separate question that has been discussed already. For example:
When is a condition variable needed, isn't a mutex enough?
Conditional Variable vs Semaphore
Advantages of using condition variables over mutex
And so on.
As has been explained, the pred parameter is optional, but having some sort of a predicate and testing it isn't. Or, in other words, not having a predicate doesn't make any sense inn a manner similar to how having a condition variable without a lock doesn't making any sense.
The reason you have a lock is because you have shared state you need to protect from simultaneous access. Some function of that shared state is the predicate.
If you don't have a predicate and you don't have a lock you really don't need a condition variable just like if you don't have a file you really don't need fwrite.
A final point is that the second code snippet you wrote is very broken. Obviously it won't compile as you define the lock after you try to pass it as an argument to condition_variable::wait(). You probably meant something like:
std::mutex mtx_cv;
std::condition_variable cv;
...
{
std::unique_lock<std::mutex> lk(mtx_cv);
cv.wait(lk);
lk.lock(); // throws std::system_error with an error code of std::errc::resource_deadlock_would_occur
}
The reason this is wrong is very simple. condition_variable::wait's effects are (from [thread.condition.condvar]):
Effects:
— Atomically calls lock.unlock() and blocks on *this.
— When unblocked, calls lock.lock() (possibly blocking on the lock), then returns.
— The function will unblock when signaled by a call to notify_one() or a call to notify_all(), or spuriously
After the return from wait() the lock is locked, and unique_lock::lock() throws an exception if it has already locked the mutex it wraps ([thread.lock.unique.locking]).
Again, why would someone want coupling waiting and locking the way std::condition_variable does is a separate question, but given that it does - you cannot, by definition, lock a std::condition_variable's std::unique_lock after std::condition_variable::wait has returned.
It's not stated in the documentation (and could be implemented differently) but conceptually you can imagine the condition variable has another mutex to both protect its own data but also coordinate the condition, waiting and notification with modification of the consumer code data (e.g. queue.size()) affecting the test.
So when you call wait(...) the following (logically) happens.
Precondition: The consumer code holds the lock (CCL) controlling the consumer condition data (CCD).
The condition is checked, if true, execution in the consumer code continues still holding the lock.
If false, it first acquires its own lock (CVL), adds the current thread to the waiting thread collection releases the consumer lock and puts itself to waiting and releases its own lock (CVL).
That final step is tricky because it needs to sleep the thread and release the CVL at the same time or in that order or in a way that threads notified just before going to wait are able to (somehow) not go to wait.
The step of acquiring the CVL before releasing the CCD is key. Any parallel thread trying to update the CCD and notify will be blocked either by the CCL or CVL. If the CCL was released before acquiring the CVL a parallel thread could acquire the CCL, change the data and then notify before the the to-be-waiting thread is added to the waiters.
A parallel thread acquires the CCL, modifies the data to make the condition true (or at least worth testing) and then notifies. Notification acquires the the CVL and identifies a blocked thread (or threads) if any to unwait. The unwaited threads then seek to acquire the CCL and may block there but won't leave wait and re-perform the test until they've acquired it.
Notification must acquire the CVL to make sure threads that have found the test false have been added to the waiters.
It's OK (possibly preferable for performance) to notify without holding the CCL because the hand-off between the CCL and CVL in the wait code is ensuring the ordering.
It may be preferrable because notifying when holding the CCL may mean all the unwaited threads just unwait to block (on the CCL) while the thread modifying the data is still holding the lock.
Notice that even if the CCD is atomic you must modify it holding the CCL or that Lock CVL, unlock CCL step won't ensure the total ordering required to make sure notifications aren't sent when threads are in the process of going to wait.
The standard only talks about atomicity of operations and another implementation may have a way of blocking notification before completing the 'add to waiters' step has completed following a failed test. The C++ Standard is careful to not dictate an implementation.
In all that, to answer some of the specific questions.
Must the state be shared? Sort of. There could be an external condition like a file being in a directory and the wait is timed to re-try after a time-period. You can decide for yourself whether you consider the file system or even just the wall-clock to be shared state.
Must there be any state? Not necessarily. A thread can wait on notification.
That could be tricky to coordinate because there has to be enough sequencing to stop the other thread notifying out of turn. The commonest solution is to have some boolean flag set by the notifying thread so the notified thread knows if it missed it. The normal use of void wait(std::unique_lock<std::mutex>& lk) is when the predicate is checked outside:
std::unique_lock<std::mutex> ulk(ccd_mutex)
while(!condition){
cv.wait(ulk);
}
Where the notifying thread uses:
{
std::lock_guard<std::mutex> guard(ccd_mutex);
condition=true;
}
cv.notify();
The reason is that in some times the waiting-thread holds the m_mutex:
#include <mutex>
#include <condition_variable>
void CMyClass::MyFunc()
{
std::unique_lock<std::mutex> guard(m_mutex);
// do something (on the protected resource)
m_condiotion.wait(guard, [this]() {return !m_bSpuriousWake; });
// do something else (on the protected resource)
guard.unluck();
// do something else than else
}
and a thread should never go to sleep while holding a m_mutex. One doesn't want to lock everybody out, while sleeping. So, atomically: {guard is unlocked and the thread go to sleep}. Once it waked up by the other-thread (m_condiotion.notify_one(), let's say) guard is locked again, and then the thread continue.
Reference (video)
Because if not so, there's a race condition before the waiting thread noticing the change of the shared state and the wait() call.
Assume we got a shared state of type std::atomic state_, there's still a fair chance for the waiting thread to miss a notification:
T1(waiting) | T2(notification)
---------------------------------------------- * ---------------------------
1) for (int i = state_; i != 0; i = state_) { |
2) | state_ = 0;
3) | cv.notify();
4) cv.wait(); |
5) }
6) // go on with the satisfied condition... |
Note that the wait() call failed to notice the latest value of state_ and may keep waiting forever.

TaskCompletionSource intermittently does not complete with NServiceBus and WCF

I have an unusual issue with TaskCompletionSource that has me baffled. I have a TaskCompletionSource waiting for the task to complete once i call the TrySetResult. I call this in three places in the code: from a WCF thread immediately to return a value to an APM WCF BeginXXX EndXXX; from another WCF thread to return immediately to the APM; lastly from an NServiceBus handler thread.
I started with the ubiquitous ToAPM provided by MS-PL. http://blogs.msdn.com/b/pfxteam/archive/2011/06/27/using-tasks-to-implement-the-apm-pattern.aspx
I noticed that the two WCF based threads worked 100% of the time. in 100 hours of hard testing, additionally extensive unit tests, I have never experienced a single failure to return a completed task to the AsyncCallback.
From the MS provided ToAPM code, the code uses a ContinueWith on the completed task to call the AsyncCallback in a schedule enabled task.
The problem I have not solved is the NServiceBus threads calling the TrySetResult on the TaskCompletionSource object. I find times of outages, where for undefined periods of time, the call simply fails. I set break points in the code for both the call and inside the ContinueWith code. I get the break point on the TrySetResult always, but only sometimes on the code inside the ContinueWith code.
The following information hopefully will shed some light on the matter.
I use a CancellationTokenSource with a timeout and setting a result to call the TrySetResult on TaskCompletionSource obj. When the above call does not work to move the task to completed, the timeout code fires. This timeout code has never not worked. it succeeds 100% of the time.
What is interesting is this, in the same code that calls the TrySetResult from the NServiceBus thread, when it works, it works as easily calling the cancellation object's Cancel as it does the TrySetResult on the TaskCompletionSource obj.
When one fails they both fail.
Then after an indiscriminate period of time it works again.
This is a WCF server in a production and QA environment and each displays identical results.
What is most weird is the following, for one WCF connection, the NServiceBus thread succeeds and another fails at the same time. Then at times both work, and then both fail. Again, all at the same time.
I have tried a number of things to work around the issue to no avail:
I wrapped the call to TrySetResult in a TaskCompletionSource + ContinueWith -- fail
I wrapped the call in a Task.Factory.StartNew -- fail
I call it directly -- fail
I really do not know what else to try.
I put in checks to ensure that the TaskCompletionSource obj is not completed, and during the outage it is not.
I put in checks to ensure the CancellationTokenSource object is not cancelled or has a cancellation pending during the outage, it does not.
I examined the objects in the debugger and they seem good.
They just do not work sometimes.
Could there be an inconsistency in the NserviceBus threads that sometimes prevent the calls from working?
Is there some thread marshaling I can try?
I searched everywhere and I have not see one mention of this problem. Is it unique?
I am totally baffled and need some ideas.
Remove the call from the NServiceBus thread execution. Isolate the call to TrySetResult using a thread such as QueueUserWorkItem or spinning your own thread. Since, the executing resumes using the thread, you may need some additional threads to handle the throughput. Ether spin multiple dedicated threads or use the thread pool. I tested calling TrySetResult in a dedicate threads and they work.
Here is code to demonstrate a single dedicated thread:
public static void Spin()
{
ClientThread = new Thread(new ThreadStart(() =>
{
while (true)
{
try
{
if (!HasSomething.WaitOne(1000, false))
continue;
while (true)
{
WaitingAsyncData entry = null;
lock (qlocker)
{
if (!Trigger.Any())
break;
entry = Trigger.Dequeue();
}
if (entry == null)
break;
entry.TrySetResult("string");
}
}
catch
{
}
}
}));
ClientThread.IsBackground = true;
ClientThread.Start();
}
Here is the ThreadPool example code:
ThreadPool.QueueUserWorkItem(delegate
{
entry.TrySetResult("string");
});
Using the ThreadPool rather than static thread provides greater flexibility and scaleability.

mutex destroyed while busy

There is a singleton object of EventHandler class to receive events from the mainthread. It registers the input to a vector and creates a thread that runs a lambda function that waits for some time before deleting the input from the vector to prevent repeated execution of the event for this input for some time.
But I'm getting mutex destroyed while busy error. I'm not sure where it happened and how it happened. I am not even sure what it meant either because it shouldn't be de-constructed ever as a singleton object. Some help would be appreciated.
class EventHandler{
public:
std::mutex simpleLock;
std::vector<UInt32> stuff;
void RegisterBlock(UInt32 input){
stuff.push_back(input);
std::thread removalCallBack([&](UInt32 input){
std::this_thread::sleep_for(std::chrono::milliseconds(200));
simpleLock.lock();
auto it = Find(stuff, input);
if (it != stuff.end())
stuff.erase(it);
simpleLock.simpleLock.unlock();
}, input)
removalCallBack.detach();
}
virtual EventResult ReceiveEvent(UInt32 input){
simpleLock.lock();
if (Find(stuff, input) != stuff.end()){
RegisterBlock(input));
//dostuff
}
simpleLock.simpleLock.unlock();
}
};
What is happening is that a thread is created
std::thread removalCallBack([&](UInt32 input){
std::this_thread::sleep_for(std::chrono::milliseconds(200));
simpleLock.lock();
...
removalCallBack.detach();
And then since removalCallBack is a local variable to the function RegisterBlock, when the function exits, the destructor for removalCallBack gets called which invokes std::terminate()
Documentation for thread destructor
~thread(); (since C++11)
Destroys the thread object. If *this still has an associated running thread (i.e. joinable() == true), std::terminate() is called.
but depending on timing, simpleLock is still owned by the thread (is busy) when the thread exits which according to the spec leads to undefined behavior, in your case the destroyed while busy error.
To avoid this error, you should either allow the thread to exist after the function exits (e.g. not make it a local variable) or block until the thread exits before the function exits using thread::join
Dealing with cleaning up after threads can be tricky especially if they are essentially used as different programs occupying the same address space, and in those cases many times a manager thread just like you thought of is created whose only job is to reclaim thread related resources. Your situation is a little easier because of the simplicity of the work done in the thread created by removalCallBack, but there still is cleanup to do.
If the thread object is going to be created by new, then although system resources used by the system thread the C++ thread object represents will get cleaned up, but the memory the object uses will remain allocated until delete is called.
Also, consider if the program exits while there are threads running, then the threads will be terminated, but if there is a mutex locked when that happens, once again there will be undefined behavior.
What is usually done to guarantee that a thread is no longer running is to join with it, but though this doesn't say, the pthread_join man page states
Once a thread has been detached, it can't be joined with pthread_join(3) or be made joinable again.

Resources