I'm struggling with multi-threaded programming...
I have an application that talks to an external device via a CAN to USB
module. I've got the application talking on the CAN bus just fine, but
there is a requirement for the application to transmit a "heartbeat"
message every second.
This sounds like a perfect time to use threads, so I created a thread
that wakes up every second and sends the heartbeat. The problem I'm
having is sharing the CAN bus interface. The heartbeat must only be sent
when the bus is idle. How do I share the resource?
Here is pseudo code showing what I have so far:
TMainThread
{
Init:
CanBusApi =new TCanBusApi;
MutexMain =CreateMutex( "CanBusApiMutexName" );
HeartbeatThread =new THeartbeatThread( CanBusApi );
Execution:
WaitForSingleObject( MutexMain );
CanBusApi->DoSomething();
ReleaseMutex( MutexMain );
}
THeartbeatThread( CanBusApi )
{
Init:
MutexHeart =CreateMutex( "CanBusApiMutexName" );
Execution:
Sleep( 1000 );
WaitForSingleObject( MutexHeart );
CanBusApi->DoHeartBeat();
ReleaseMutex( MutexHeart );
}
The problem I'm seeing is that when DoHeartBeat is called, it causes the
main thread to block while waiting for MutexMain as expected, but
DoHeartBeat also stops. DoHeartBeat doesn't complete until after
WaitForSingleObject(MutexMain) times out in failure.
Does DoHeartBeat execute in the context of the MainThread or
HeartBeatThread? It seems to be executing in MainThread.
What am I doing wrong? Is there a better way?
Thanks,
David
I suspect that the CAN bus API is single-threaded under the covers. It may be marshaling your DoHeartBeat() request from your second thread back to the main thread. In that case, there would be no way for it to succeed since your main thread is blocked. You can fix this in basically two ways: (1) send a message to the main thread, telling it to do the heart beat, rather than doing it on the second thread; or (2) use a timer on the main thread for your heart beat instead of a second thread. (I do think that multithreading is overkill for this particular problem.)
First, re-read the specs about the heartbeat. Does it say that an actual heartbeat message must be received every second, or is it necessary that some message be received every second, and that a heartbeat should be used if no other messages are in flight? The presence of data on the channel is de-facto evidence that the communications channel is alive, so no specific heartbeat message should be required.
If an actual heartbeat message is required, and it's required every second, in the above code there should be only one mutex and both threads need to share it. The code as written creates two separate mutexes, so neither will actually block. You'll end up with a collision on the channel and Bad Things Will Happen in CanBusApi. Make MainMutex visible a global/class variable and have both threads reference it.
Related
I am new to Rust and trying to understand the Dining Philosopher code here :
https://google.github.io/comprehensive-rust/exercises/day-4/solutions-morning.html
By the time the execution reaches the following lines in the main thread, isn't it possible that none of the spawned threads have started executing their logic, resulting in nothing in 'rx' and the program simply quitting?
for thought in rx {
println!("{}", thought);
}
When iterating over a channel, it internally calls Receiver::recv, where the documentation specifies
This function will always block the current thread if there is no data available and it’s possible for more data to be sent (at least one sender still exists). Once a message is sent to the corresponding Sender (or SyncSender), this receiver will wake up and return that message.
So the receiver will block until it has data avalible, or all the senders have been dropped.
Yes, execution can reach for thought in rx { ... } before the threads have even started. However, this will still work because iterating over a Receiver will wait until there is a message and will only stop if all Senders have been destroyed (ergo it is no longer possible to receive any messages).
This is using a sub-classed QThread based on the ideas expressed in the whitepaper "QThreads: You were not doing so wrong". It does not have an event loop, nor does it have slots. It just emits signals and stops. In fact its primary signal is the QThread finished one.
Basically I have a Qt using a background thread to monitor stuff. Upon finding what it is looking for, it records its data, and terminates.
The termination sends a signal to the main event loop part of the application, which processes it, and when done, starts the background anew. I can usually get this working for tens of seconds, but then it just seems to quit.
It seems that when the main application tries to start the thread, it doesn't really run. I base this on telemetry code that increments counters as procedures get executed.
basically
//in main application. Setup not shown.
//background points to the QThread sub-class object
void MainWindow::StartBackground()
{
background->startcount++;
background->start();
if ( background->isRunning() )
{
background->startedcount++;
}
}
//in sub-classed QThread
void Background::run()
{
runcount++;
//Do stuff until done
}
So when I notice that it seems that my background thread isn't running, by watching Process Explorer, I cause the debugger to break in, and check the counts. What I see is that startcount and startedcount are equal. And have a value of one greater than runcount
So I can only conclude that the thread didn't really run, but I have been unable to find out any evidence of why.
I have not been able to find documentation on QThreads not starting do to some error condition, or what evidence there is of such an error.
I suppose I could set up a slot to catch started from the thread. The starting code could loop on a timed-out semaphore, trying again and again until the started slot actually resets the semaphore. But it feels ugly.
EDIT - further information
So using the semaphore method, I have a way to breakpoint on failure to start.
I sampled isFinished() right before I wanted to do start(), and it was false. After my 100ms semaphore timeout it became true.
So the question seems to be evolving into 'Why does QThread sometimes emit a finished() signal before isFinished() becomes true?'
Heck of a race condition. I'd hate to spin on isFinished() before starting the next background thread.
So this may be a duplicate of
QThread emits finished() signal but isRunning() returns true and isFinished() returns false
But not exactly, because I do override run() and I have no event loop.
In particular the events 8 and 9 in that answer are not in the same order. My slot is getting a finished() before isFinished() goes true.
I'm not sure an explicit quit() is any different than letting run() return;
It sounds as if you have a race condition wherein you may end up trying to restart your thread before the previous iteration has actually finished. If that's the case then, from what I've seen, the next call to QThread::start will be silently ignored. You need to update your code so that it checks the status of the thread before restarting -- either by calling QThread::isFinished or handling the QThread::finished signal.
On the other hand... why have the thread repeatedly started/stopped. Would it not be easier to simply start the thread once? Whatever code is run within the context of QThread::run can monitor whatever it monitors and signal the main app when it finds anything of note.
Better still. Separate the monitor logic from the thread entirely...
class monitor: public QObject {
.
.
.
};
QThread monitor_thread;
monitor monitor;
/*
* Fix up any signals to/from monitor.
*/
monitor.moveToThread(&monitor_thread);
monitor_thread.start();
The monitor class can do whatever it wants and when it's time to quit the app can just call monitor_thread::quit.
There is a race condition in the version of Qt I am using. I don't know if it was reported or not before, but I do not have the latest, so it's probably moot unless I can demonstrate it in the current version.
Similar bugs were reported here long ago:
QThread.isFinished returns False in slot connected to finished() signal
(the version I use is much more recent than Qt 4.8.5)
What more important is I can workaround it with the following code
while ( isRunning() )
{
msleep(1);
}
start();
I've run a few tests, and it never seems to take more than 1ms for the race condition to settle. Probably just needs a context switch to clean up.
I'm using the Azure Service Bus SubscriptionClient.OnMessage method; configured to process up to 5 messages concurrently.
Within the code I need to wait for all messages to finish processing before I can continue (to properly shutdown an Azure Worker Role). How do I do this?
Will SubscriptionClient.Close() block until all messages have finished processing?
Calling Close on SubscriptionClient or QueueClient will not block. Calling Close closes off the entity immediately as far as I can tell. I tested quickly just using the Worker Role With Service Bus Queue project template that shipped with Windows Azure SDK 2.0. I added a thread sleep for many seconds in the message process action and then shut down the role while it was running. I saw the Close method get called while the messages were processing in their thread sleep but it certainly did not wait for the for message processing to complete, the role simple closed down.
To handle this gracefully you'll need to do the same thing we did when dealing with any worker role that was processing messages (Service Bus, Azure Storage queue or anything else): keep track of what is being worked on and shut down when it is complete. There are several ways to deal with that but all of them are manual and made messy in this case because of the multiple threads involved.
Given the way that OnMessage works you'll need to add something in the action that looks to see if the role has been told to shutdown, and if so, to not do any processing. The problem is, when the OnMessage action is executed it HAS a message already. You'd probably need to abandon the message but not exit the OnMessage action, otherwise it will keep getting a message if there are ones in the queue. You can't simply abandon the message and let the execution leave the action because then the system will be handed another message (possibly the same one) and several threads doing this may cause messages to get too many dequeue counts and get dead lettered. Also, you can't call Close on the SubscriptionClient or QueueClient, which would stop the receive loop internally, because once you call close any of the outstanding message processing will throw an exception when .Complete, .Abandon, etc. is called on the message because the message entity is now closed. This means you can't stop the incoming messages easily.
The main issue here is because you are using the OnMessage and setting up the concurrent message handling by setting the MaxConcurrentCalls on the OnMessageOptions, which means the code that starts and manages the threads is buried in the QueueClient and SubscriptionClient and you don't have control over that. You don't have a way to reduce the count of threads, or stop the threads individually, etc. You'll need to create a way to put the OnMessage action threads into a state where they are aware that the system is being told to shut down and then complete their message and not exit the action in order for them to not continuously be assigned new messages. This means you'll likely need to also set the MessageOptions to not use autocomplete and manually call complete in your OnMessage action.
Having to do all of this may severely reduce the actual benefit of using the OnMessage helper. Behind the scenes OnMessage is simply setting up a loop calling receive with the default timeout and handing of messages to another thread to do the action (loose description). So what you get by using the OnMessage approach is away from having to write that handler on your own, but then the problem you are having is because you didn't write that handler on your own you don't have control over those threads. Catch-22. If you really need to stop gracefully you may want to step away from the OnMessage approach, write your own Receive loop with threading and within the main loop stop receiving new messages and wait for all the workers to end.
One option, especially if the messages are idempotent (which means processing them more than once yields the same results... which you should be mindful of anyway) then if they are stopped in mid processing they will simply reappear on the queue to be processed by another instance later. If the work itself isn't resource intensive and the operations are idempotent then this really can be an option. No different than when an instance might fail due to hardware failure or other issues. Sure, it's not graceful or elegant, but it certainly removes all the complexity I've mentioned and is still something that can happen anyway due to other failures.
Note that the OnStop is called when an instance is told to shut down. You've got 5 minutes you can delay this until the fabric just shuts it off, so if your messages take longer than five minutes to process it won't really matter if you attempt to shut down gracefully or not, some will be cut off during processing.
You can tweak OnMessageAsync to wait for processing of messages to complete, and block new messages from beginning to be processed:
Here is the implementation:
_subscriptionClient.OnMessageAsync(async message =>
{
if (_stopRequested)
{
// Block processing of new messages. We want to wait for old messages to complete and exit.
await Task.Delay(_waitForExecutionCompletionTimeout);
}
else
{
try
{
// Track executing messages
_activeTaskCollection[message.MessageId] = message;
await messageHandler(message);
await message.CompleteAsync();
}
catch (Exception e)
{
// handle error by disposing or doing nothing to force a retry
}
finally
{
BrokeredMessage savedMessage;
if (!_activeTaskCollection.TryRemove(message.MessageId, out savedMessage))
{
_logger.LogWarning("Attempt to remove message id {0} failed.", savedMessage.MessageId);
}
}
}
}, onMessageOptions);
And an implementation of Stop that waits for completion:
public async Task Stop()
{
_stopRequested = true;
DateTime startWaitTime = DateTime.UtcNow;
while (DateTime.UtcNow - startWaitTime < _waitForExecutionCompletionTimeout && _activeTaskCollection.Count > 0)
{
await Task.Delay(_waitForExecutionCompletionSleepBetweenIterations);
}
await _subscriptionClient.CloseAsync();
}
Note that _activeTaskCollection is a ConcurrentDictionary (we can also use a counter with interlock to count the number of in progress messages, but using a dictionary allows you to investigate what happend easily in case of errors.
I'm building a server with NIO, I have two questions.
Do I have to use a worker thread or a thread pool to process the messages received, or let the main thread do all this stuff ( I have performance needs).
I have two kind of sending, sendNow method which ends with selector.selectNow() and simple send method which ends with selector.wakeup().. can I have loss of data those methods?
thanks
If possible try to do it all in one thread. It gets very complicated very quickly otherwise.
I don't know why you think a sendNow() method needs to end with either selectNow() or wakeup(), but neither of them is intrinsically going to cause a data loss.
Mfc provides both worker and UI thread. UI thread is enabled with message receiving capabilities (send, post). Could it be possible to let worker thread too receive messages.
Call CWinThread::PumpMessage() repeatedly until it returns a WM_QUIT message.
It seems you need a thread, that can handle multiple messages from another threads. Another threads would add-a-message to the message-queue of this thread. Well, in that case you may use PeekMessage to startup a loop, which would eventually create a hidden window, and then use GetMessage to get the messages. The other threads would use PostThreadMessage with the thread ID (the one having Peek/GetMessage), and the message-code, LPARAM, WPARAM.
It would be like (not syntactically correct):
TheProcessor()
{
MSG msg;
PeekMessage(&msg,...);
while(GetMessage(&msg...)
{ /* switch case here */ }
}
The threads would call PostThreadMessage - See MSDN for more info.
When you need to send more data than LPARAM/WPARAM can hold, you eventually need to allocate them on heap, and then delete AFTER processing the message in your custom message-loop. This would be cumbersome and buggy.
But... I would suggest you to have your own class, on top of std::queue/deque or other DS, where you can add AddMessage/PushMessage, and PopMessage (or whatever names you like). You need to use SetEvent, WaitForSingleObject to trigger the new message in loop (See one of the implementation here. You may make it generic for one data-type, or make it template class - that would support any data-type (your underlying DS (queue) would utilize the same data-type). You also need not to worry about heaps and deletions. This is less error prone. You may however, have to handle MT issues.
Using Windows events involves kernel mode transition (since events are named/kernel objects), and you may like to use Conditional Variables which are user objects.Or you may straightaway use unbounded_buffer class from Concurrency Runtime Library available in VC10. See this article (jump to unbounded_buffer).
Yes you can create a message queue on a worker thread. You will need to run a message pump on that thread.