Short version: on OS X, if I call VideoCapture::read() from a thread other than the main() thread, the call returns immediately instead of blocking till a new frame.
This one works as expect:
void main()
{
VideoCapture vc(0);
Mat img;
While(1) {
vc.read( img ); // blocks till new frame arrives
}
}
This one does not block:
void run( VideoCapture& vc )
{
Mat img;
While(1) {
vc.read( img ); // returns immediately and always returns true
}
}
void main()
{
VideoCapture vc(0);
boost::thread capThread( boost::bind( &run, vc ) );
capThread.join();
}
So in the second version with a separate thread to grab frames, the call to VideoCapture::read(img) returns immediately with return value true, and img is set to the current frame, which means it'll return many duplicate frames.
It's mentioned here:
Problem accessing camera when using Boost thread on OSX
that:
"The OpenCV Camera functions on mac require access to an objective-c NSRunLoop; don't know how to get at one from a new thread though."
Anyone know of a solution to do a blocking frame grab from a thread other than the main() thread?
Alternatively, is there a nice way to discard the duplicate frames?
Thanks
OpenCV doesn't support multi-threading.
Let the main thread capture the frames and do whatever you need to do in the second thread.
Related
I am trying to synchronize threads writing data to a text file in a class by using Monitor, but in my code it seems that the else statement is never evaluated, is this the correct use of monitor for thread synchronization?
void Bank::updatefile()
{
Thread^ current = Thread::CurrentThread;
bool open = false;
current->Sleep(1000);
while (!open)
{
if (Monitor::TryEnter(current))
{
String^ fileName = "accountdata.txt";
StreamWriter^ sw = gcnew StreamWriter(fileName);
for (int x = 0; x < 19; x++)
sw->WriteLine(accountData[x]);
sw->Close();
Monitor::Pulse;
Monitor::Exit(current);
current->Sleep(500);
open = true;
}
else
{
Monitor::Wait(current);
current->Sleep(500);
}
}
}
You're passing an object to Monitor::TryEnter that is specific to the thread in which it is executing (i.e. Thread^ current = Thread::CurrentThread;). No other threads are using the same object (they're using the one for their own thread). So there's never a collision or locking conflict.
Try creating some generic object that's shared among the threads, something higher up in the Bank class. Then use that for your TryEnter call.
Your use of Monitor is partially correct. The object you're using for locking is not correct.
Monitor
Monitor::Pulse is unnecessary here. Just Exit the monitor, and the next thread will be able to grab the lock.
Monitor::Wait is incorrect here: Wait is supposed to be used when the thread has the object already locked. Here, you have the object not locked yet.
In general, Pulse and Wait are rarely used. For locking for exclusive access to a shared resource, Enter, TryEnter, and Exit are all you need.
Here's how your use of Monitor should be written:
Object^ lockObj = ...;
bool done = false;
while(!done)
{
if(Monitor::TryEnter(lockObj, 500)) // wait 500 millis for the lock.
{
try
{
// do work
done = true;
}
finally
{
Monitor::Exit(lockObj);
}
}
else
{
// Check some other exit condition?
}
}
or if the else is empty, you can simplify it like this:
Object^ lockObj = ...;
Monitor::Enter(lockObj); // Wait forever for the lock.
try
{
// do work
}
finally
{
Monitor::Exit(lockObj);
}
There is a class that Microsoft provides that makes this all easier: msclr::lock. This class, used without the ^, makes use of the destructor to release the lock, without a try-finally block.
#include <msclr\lock.h>
bool done = false;
while(!done)
{
msclr::lock lock(lockObj, lock_later);
if (lock.try_acquire(500)) // wait 500 millis for the lock to be available.
{
// Do work
done = true;
}
} // <-- Monitor::Exit is called by lock class when it goes out of scope.
{
msclr::lock lock(lockObj); // wait forever for the lock to be available.
// Do work
} // <-- Monitor::Exit is called by lock class when it goes out of scope.
The object to lock on
Thread::CurrentThread is going to return a different object on each thread. Therefore, each thread attempts to lock on a different object, and that's why all of them succeed. Instead, have one object, created before you spawn your threads, that is used for locking.
Also, instead of opening & closing the file on each thread, it would be more efficient to open it once, before the threads are spawned, and then just use that one StreamWriter from each of the threads. This also gives you a obvious object to lock on: You can pass the StreamWriter itself to Monitor::Enter or msclr::lock.
I'm trying to use threads to manage several things in GTK+, however, as soon as I try to use any GUI function in the new thread, it locks up the GUI and this makes sense since GTK+ is not thread safe. Is there anyway around this?
Here's my code:
int main(int argc, char *argv[])
{
GError *error = NULL;
/* init threads */
g_thread_init(NULL);
gdk_threads_init();
/* init gtk */
gtk_init(&argc, &argv);
....
//Multithreaded functions
g_thread_create(argument_thread, (gpointer)label7, FALSE, &error );
gdk_threads_enter();
gtk_main();
gdk_threads_leave();
return 0;
}
void *argument_thread(void *args)
{
while(1)
{
gdk_threads_enter();
gtk_entry_set_text(entry2,"random stuff");
gdk_threads_leave();
}
}
Not sure if this could be an issue (don't know GTK) but maybe there is a race condition if the thread acquires the lock before the gtk_main has started.
Then you could try:
gdk_threads_enter();
//Multithreaded functions
g_thread_create(argument_thread, (gpointer)label7, FALSE, &error );
gtk_main();
gdk_threads_leave();
Moreover you should temporize your loop:
void *argument_thread(void *args)
{
while(1)
{
gdk_threads_enter();
gtk_entry_set_text(entry2,"random stuff");
gdk_threads_leave();
sleep(10);
}
}
I have resolved the problem using g_timeout e gthread:http://www.freemedialab.org/wiki/doku.php?id=programmazione:gtk:gtk_e_i_thread
Basically I use 3 functions, one that launches the thread, one that does the job without manipulating widgets (thread) and a third type that serves as a timeout timer checking every n seconds certain values written by the thread and updates the ' graphic interface.
Or you can use "g_idle_add" : http://www.freemedialab.org/wiki/doku.php?id=programmazione:gtk:gtk_e_i_thread#versione_con_g_idle_add
gdk_threads_enter() and gdk_threads_leave() are deprecated from 3.6 version of Gtk.
I'm currently working on an application that needs to utilize a form to block the user from using the main form for a few seconds. Although running the code on the main thread seems to be okay, the labels on the pop up form don't render for about a second when the form first appears. I thought that if I ran that form on a separate thread, the rendering would be a lot smoother. The rendering is now so smooth that the form disappears immediately after rendering. The timer is set for five seconds, with a label on the screen counting down. Here's the relevant code calling the new thread and form:
System::Void MainScreen::runGame(int playerTurn) {
Thread ^ t = gcnew Thread(gcnew ThreadStart(gcnew MainScreen(),
&MainScreen::showModalDialog));
t->Start();
t->Join();
InitializeDice();
startTimer();
}
System::Void MainScreen::showModalDialog() {
GetReady ^ gr = gcnew GetReady();
gr->showModalPopup();
}
And here's the code inside the form:
public:
GetReady(void)
{
InitializeComponent();
}
System::Void showModalPopup() {
this->Show();
startTimer();
}
private: System::Void timerPrep_Tick(System::Object^ sender, System::EventArgs^ e) {
ts = ts->Subtract(TimeSpan(0, 0, 1));
if (ts->TotalSeconds <= 0) {
finishTimer();
} else {
lblTimer->Text = ts->ToString("mm\\:ss");
}
}
System::Void startTimer() {
array<String ^>^ minSec = gcnew array<String ^>(2);
minSec = lblTimer->Text->Split(':');
ts = gcnew TimeSpan(0, Convert::ToInt32(minSec[0]), Convert::ToInt32(minSec[1]));
Thread::Sleep(900);
timerPrep->Start();
}
System::Void finishTimer() {
timerPrep->Stop();
lblTimer->Text = "GO!!!";
Thread::Sleep(900);
this->Close();
}
My ideal solution would be to use a thread to generate the new form, so that rendering in both the main form and the pop up form is smooth.
Things I've tried:
Moving this->Show() every where I can think to put it.
I've added t->Join() to see if the main thread was trying to bring the main window back into focus. Thread t still executes and the timer still runs appropriately with the Join, blocking user input for five seconds - but there is nothing to block the screen.
I've read a few questions on SO, I think the most relevant one is WinForms multithreading issue - although I feel like that might be overkill for this situation.
If you have any ideas on what I need to do to have smooth rendering for both forms, please let me know. Thanks!
Displaying UI on a worker thread is filled with traps and surprises. The chief problem you are having here is that the thread doesn't pump a message loop. Required first of all to ensure that the window can receive Windows messages and to ensure that the thread doesn't immediately terminate. Beyond Jairo's recommendation to use ShowDialog(), the boilerplate solution is:
System::Void MainScreen::showModalDialog() {
Application::Run(gcnew GetReady);
}
There's more, a thread that displays windows should always be an STA thread. An implementation detail for COM and required to get stuff like the clipboard, drag+drop and shell dialogs operating correctly:
Thread ^ t = gcnew Thread(gcnew ThreadStart(this, &MainScreen::showModalDialog));
t->SetApartmentState(ApartmentState::STA);
t->Start();
And you generally have a Z-order problem with windows that are displayed on the main thread. With a nasty habit of your window disappearing behind a window owned by the main thread. No good solution for that.
The problem is showModalPopup uses Show method instead ShowDialog. The thread starts, shows the form (it doesnt block execution), starts the timer, ends and joins to the main thread. The thread where the form was created has been finished and it's here where the winforms multithreading issues comes.
Start the timer first and then show the modal window with ShowDialog. Something like this:
System::Void showModalPopup()
{
startTimer();
this->ShowDialog();
}
I'm working on a Qt application to control an industrial camera, and in particular I need to trigger the camera at a particular time (when various illumination settings have been put in place, for example), and wait until a frame is returned. In the simplest case, the following code does the job nicely:
void AcquireFrame()
{
// Runs in the main GUI thread:
camera -> m_mutex.lock();
camera -> frameHasArrived = false;
camera -> m_mutex.unlock();
camera -> triggerImageAcquisition();
forever
{
camera -> m_mutex.lock()
bool isReady = camera -> frameHasArrived;
camera -> m_mutex.unlock()
if (isReady)
{
return;
}
else
{
Sleep(10);
}
}
}
void callback(camera *)
{
// Called by the camera driver from a separate OS thread - not a Qt thread -
// when a frame is ready:
camera -> m_mutex.lock();
camera -> frameHasArrived = true;
camera -> m_mutex.unlock();
}
...and most of the time this works perfectly well. However, this being the real world, occasionally the camera will fail to receive the trigger or the computer will fail to receive the frame cleanly, and the above code will then go into an infinite loop.
The obvious thing to do is to put in a timeout, so if the frame is not received within a certain time then the image acquisition can be attempted again. The revised code looks like:
void AcquireFrame()
{
camera -> m_mutex.lock();
camera -> frameHasArrived = false;
camera -> m_mutex.unlock();
camera -> triggerImageAcquisition();
QTime timeout;
timeout.start();
forever
{
timeout.restart();
fetch: camera -> m_mutex.lock()
bool isReady = camera -> frameHasArrived;
camera -> m_mutex.unlock()
if (isReady)
{
return;
}
else if (timeout.elapsed() > CAM_TIMEOUT)
{
// Assume the first trigger failed, so try again:
camera -> triggerImageAcquisition();
continue;
}
else
{
Sleep(10);
goto fetch;
}
}
}
Now, the problem is that with this latter version the failure rate (the proportion of 'unsuccessful triggers') is much, much higher - at least an order of magnitude. Moreover, this code too will eventually find itself in an infinite loop where, however many times it tries to re-trigger the camera, it never sees a frame come back. Under these latter circumstances, killing the application and checking the camera reveals that the camera is in perfect working order and patiently waiting for its next trigger, so it doesn't appear to be a camera problem. I'm coming to the conclusion that in fact it's some sort of a system resource issue or a thread conflict, so that Qt's event loop is not allowing the camera callback to be called at the proper time.
Is this likely, and is there in fact a better way of doing this?
Update on 6th June:
For what it's worth, I've seen no more problems since I adopted the method below (having given the camera object an extra member, namely a QWaitCondition called 'm_condition'):
void AcquireFrame()
{
bool frameReceived;
forever
{
camera -> triggerImageAcquisition();
camera -> m_mutex.lock();
frameReceived = camera -> m_condition.wait(&camera->m_mutex, CAM_TIMEOUT);
if (frameReceived)
{
// We received a frame from the camera, so can return:
camera -> m_mutex.unlock();
return;
}
// If we got to here, then the wait condition must have timed out. We need to
// unlock the mutex, go back to the beginning of the 'forever' loop and try
// again:
camera -> m_mutex.unlock();
}
}
void callback (camera *)
{
// Called by the camera driver from a separate OS thread -
// not a QThread - when a frame is ready:
camera -> m_condition.wakeOne();
}
This still has the effect of pausing the main thread until we have either received a frame or experienced a timeout, but now we have eliminated the Sleep() and the Qt event loop remains in full control throughout. It's still not clear to me why the old method caused so many problems - I still suspect some sort of system resource limitation - but this new approach seems to be more lightweight and certainly works better.
Running the AcquireFrame that blocks on mutexes in the GUI thread makes not much sense to me unless you wanted to trade off GUI reponsiveness for latency, but I doubt that you care about the latency here as the camera snaps single frames and you insist on processing them in the busy GUI thread in the first place.
Secondly, there is nothing that Qt would do to prevent the callback from getting called from the other thread, other than the other thread having lower priority and being preempted by higher priority threads completely monopolizing the CPU.
I would simply post an event to a QObject in the GUI thread (or any other QThread!) from the callback function. You can post events from any thread, it doesn't matter -- what matters is the receiver. QCoreApplication::postEvent is a static function, after all, and it doesn't check the current thread at all.
In a complex application you probably want to have the logic in a dedicated controller QObject, and not spread across QWidget-derived classes. So you'd simply post the event to the controller instance.
Do note that posting an event to an idle GUI thread will work exactly the same as using a mutex -- Qt's event loop uses a mutex and sleeps on that mutex and on messages from the OS. The beautiful thing is that Qt already does all the waiting for you, but the wait is interruptible. The posted event should have a high priority, so that it'll end up the first one in the queue and preempt all the other events. When you're ready to acquire the frame, but before you trigger it, you can probably call QCoreApplication::flush(). That's about it.
There should be no problem in having a separate image processor QObject living in a dedicated QThread to leverage multicore machines. You can then process the image into a QImage, and forward that one to the GUI thread using another event, or simply via a signal-slot connection. You can probably also notify the GUI thread when you've acquired the frame but are only beginning to process it. That way it'd be more obvious to the user that something is happening. If image processing takes long, you can even send periodic updates that'd be mapped to a progress bar.
The benchmark results (using release builds) are interesting but in line with the fact that Qt's event queues are internally protected by a mutex, and the event loop waits on that mutex. Oh, the results seem to be portable among mac and windows xp platforms.
Using a naked wait condition is not the fastest way, but using a naked posted event is even slower. The fastest way is to use a queued signal-slot connection. In that case, the cost of posting an event to the same thread (that's what the FrameProcessorEvents::tick() does) seems to be negligible.
Mac
warming caches...
benchmarking...
wait condition latency: avg=45us, max=152us, min=8us, n=1001
queued signal connection latency: avg=25us, max=82us, min=10us, n=1000
queued event latency: avg=71us, max=205us, min=14us, n=1000
Windows XP under VMWare Fusion
Note that results over 1ms are likely due to VMWare being not scheduled at the moment.
warming caches...
benchmarking...
wait condition latency: avg=93us, max=783us, min=8us, n=1000
queued signal connection latency: avg=46us, max=1799us, min=0us, n=1000
queued event latency: avg=117us, max=989us, min=18us, n=1001
Below is the benchmarking code.
#include <cstdio>
#include <limits>
#include <QtCore>
QTextStream out(stdout);
class TimedBase : public QObject
{
public:
TimedBase(QObject * parent = 0) : QObject(parent) { reset(); }
friend QTextStream & operator<<(QTextStream & str, const TimedBase & tb) {
return str << "avg=" << tb.avg() << "us, max=" << tb.usMax << "us, min="
<< tb.usMin << "us, n=" << tb.n;
}
void reset() { usMax = 0; n = 0; usMin = std::numeric_limits<quint32>::max(); usSum = 0; }
protected:
quint64 n, usMax, usMin, usSum;
quint64 avg() const { return (n) ? usSum/n : 0; }
void tock() {
const quint64 t = elapsed.nsecsElapsed() / 1000;
usSum += t;
if (t > usMax) usMax = t;
if (t < usMin) usMin = t;
n ++;
}
QElapsedTimer elapsed;
};
class FrameProcessorEvents : public TimedBase
{
Q_OBJECT
public:
FrameProcessorEvents(QObject * parent = 0) : TimedBase(parent) {}
public slots: // can be invoked either from object thread or from the caller thread
void tick() {
elapsed.start();
QCoreApplication::postEvent(this, new QEvent(QEvent::User), 1000);
}
protected:
void customEvent(QEvent * ev) { if (ev->type() == QEvent::User) tock(); }
};
class FrameProcessorWait : public TimedBase
{
Q_OBJECT
public:
FrameProcessorWait(QObject * parent = 0) : TimedBase(parent) {}
void start() {
QTimer::singleShot(0, this, SLOT(spinner()));
}
public: // not a slot since it must be always invoked in the caller thread
void tick() { elapsed.start(); wc.wakeAll(); }
protected:
QMutex mutex;
QWaitCondition wc;
protected slots:
void spinner() {
forever {
QMutexLocker lock(&mutex);
if (wc.wait(&mutex, 1000)) {
tock();
} else {
return;
}
}
}
};
FrameProcessorEvents * fpe;
FrameProcessorWait * fpw;
static const int avgCount = 1000;
static const int period = 5;
class FrameSender : public QObject
{
Q_OBJECT
public:
FrameSender(QObject * parent = 0) : QObject(parent), n(0), N(1) {
QTimer::singleShot(0, this, SLOT(start()));
}
protected slots:
void start() {
out << (N ? "warming caches..." : "benchmarking...") << endl;
// fire off a bunch of wait ticks
n = avgCount;
timer.disconnect();
connect(&timer, SIGNAL(timeout()), SLOT(waitTick()));
fpw->reset();
fpw->start();
timer.start(period);
}
void waitTick() {
fpw->tick();
if (!n--) {
if (!N) { out << "wait condition latency: " << *fpw << endl; }
// fire off a bunch of signal+event ticks
n = avgCount;
fpe->reset();
timer.disconnect();
connect(&timer, SIGNAL(timeout()), fpe, SLOT(tick()));
connect(&timer, SIGNAL(timeout()), SLOT(signalTick()));
}
}
void signalTick() {
if (!n--) {
if (!N) { out << "queued signal connection latency: " << *fpe << endl; }
// fire off a bunch of event-only ticks
n = avgCount;
fpe->reset();
timer.disconnect();
connect(&timer, SIGNAL(timeout()), SLOT(eventTick()));
}
}
void eventTick() {
fpe->tick();
if (!n--) {
if (!N) { out << "queued event latency: " << *fpe << endl; }
if (!N--) {
qApp->exit();
} else {
start();
}
}
}
protected:
QTimer timer;
int n, N;
};
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QThread eThread;
QThread wThread;
eThread.start(QThread::TimeCriticalPriority);
wThread.start(QThread::TimeCriticalPriority);
fpw = new FrameProcessorWait();
fpe = new FrameProcessorEvents();
fpw->moveToThread(&eThread);
fpe->moveToThread(&wThread);
FrameSender s;
a.exec();
eThread.exit();
wThread.exit();
eThread.wait();
wThread.wait();
return 0;
}
#include "main.moc"
How much work is it to detect the trigger state and fire the camera?
If that's relatively cheap - I would have a separate thread just blocking on a trigger event and firing the camera. Then have the main thread informed by a Qt signal sent from the callback function.
I think I have a problem in my program.
I must create an object that continuosly communicate with an external tracking system and get coordinates of point from it.
I wrapped this class inside a boost::thread and before the first calls to my Glut Application I create the thread object and I detach it
The code for the salient methods of the class is the following
boost::mutex resourceMutex;
void Tracker::init()
{
boost::mutex::scoped_lock lock(resourceMutex);
try
{
// some initializations
}
catch (std::bad_alloc const&)
{
cerr << "Memory allocation fail during init!" << endl;
}
try
{
p3dData = (Position3d*)calloc( NUM_MARKERS , sizeof( Position3d ) );
if ( p3dData==NULL )
throw std::bad_alloc();
}
catch ( std::bad_alloc const&)
{
cerr << "Memory allocation fail during memory allocation!" << endl;
}
}
void Tracker::update()
{
boost::mutex::scoped_lock lock(optotrakResourceMutex);
//... operations on vector< Eigen::Vector3d > points
}
vector<Eigen::Vector3d> &Tracker::getAllPoints()
{
return points;
}
My glutTimerFunc makes a call to an update function that every frame picks the points with the method getAllPoints, while the tracker thread continuosly update them (in fact the frequencies of access to data are different, the thread calls to is faster than the glut update functions calls.
Now when the program exit, I first delete the Tracker object allocated with new then interrupt the thread containing it, but sometimes I get strange behaviours I think they are memory leak
Is the way of getting data with different frequencies of access and the use of scoped_lock correct or should I put some guard in the getAllPoints method?
I understand your dedicated tracker thread continuously calls Tracker::update() to acquire the localization data from your device (NDI Optotrak?)
Then, your OpenGL application accesses the latest points at regular interval from the main thread using Tracker::getAllPoints().
In this case, the vector of 3D points Tracker::points is a shared resource between these two threads.
To prevent concurrent access, both the writing operation in update() and the reading with getAllPoints() must be protected by the mutex, not only the writing as in your current code. The reading code in the main thread must also lock the mutex:
// In your main application:
void timerFunc()
{
Tracker* tracker = ...; // Obtain a pointer to the tracker object
tracker->LockResourceMutex(); // Enter critical section
vector< Eigen::Vector3d >& pointsRef = tracker->getAllPoints();
//... operations on points, protected by the mutex
tracker->UnlockResourceMutex(); // Leave critical section
}
// In class Tracker:
void Tracker::LockResourceMutex() { optotrakResourceMutex.lock(); }
void Tracker::UnlockResourceMutex() { optotrakResourceMutex.unlock(); }
Caveat: If your operations on points in the timerFunc() are slow, then the mutex will remain locked for a long time and your tracker thread will block on it when calling Tracker::update().
A better design would be to change Tracker::getAllPoints() to return a copy of the 3D points vector instead of a reference:
// In class Tracker:
vector<Eigen::Vector3d> Tracker::getAllPoints()
{
boost::mutex::scoped_lock lock(optotrakResourceMutex);
return points; // Will call the std::vector() copy constructor
}
// In your main application:
void timerFunc()
{
Tracker* tracker = ...; // Obtain a pointer to the tracker object
vector< Eigen::Vector3d > myPoints = tracker->getAllPoints();
//... operations on your own copy if points
}
Note how the mutex is encapsulated in the Tracker class and how the timerFunc() does not need to worry about it.
Also note how the mutex is locked only during the copy. The copy of a list of 3D vectors is certainly going to be faster than mathematical operations on them.