Semaphore based synchronization for opencv based image processing - multithreading

I am trying to use semaphores to synchronize two threads of OpenCV image processes. One thread keeps capturing frames from the webcam and pushes it into a circular buffer and the other stream pulls an image from the buffer and processes them. I am deleting the frame that I use after I process them.
I am using boost circular buffer libraries to implement the buffer.
Please find the exact code snippet below. I have eliminated most part of the initialization and highlighting thats necessary. The problem that I am facing is that I am able to stream and capture my webcam's frames. But the window just freezes after sometimes. On some occasions, the captured frames are not displayed correctly. May I know where am I going wrong?
sem_t semaphore;
using namespace cvb;
using namespace std;
CvCapture* capture = cvCaptureFromCAM( CV_CAP_ANY );
IplImage* img0;
IplImage* img1;
sem_t semaphore;
boost::circular_buffer<IplImage*> cb1(200);
int ret = sem_init(&semaphore, 1, 10); //Initializing the semaphore
void* Capture(void* arg){
while(1) {
sem_wait(&semaphore);
img0 = cvQueryFrame( capture );
cb1.push_back(img0);
cvShowImage( "mywindow", img0 );
sem_post(&semaphore);
if ( (cvWaitKey(10) & 255) == 27 ) break;
}
}
void* ImageProcessing(void* arg) {
while(1) {
if(!cb1.empty()) {
sem_wait(&semaphore);
img2 = cvShowImage("result",img2);
cb1.pop_front();
sem_post(&semaphore);
}
if ( (cvWaitKey(10) & 255) == 27 ) break;
}
cvReleaseCapture( &capture );
cvReleaseImage( &img2 );
}

Capture and ImageProcessing are being run in different threads, aren't they? I once tried to update HighGui windows from different threads, but it didn't work.
This is actually a problem with most if not all windowing systems: you can't make calls from different threads to update a window.
Try putting both cvShowImage calls in the same thread. Calls to cvWaitKey probably have to be made from within the same thread too.
It may be the case that cvShowImage must be called in the same thread in which you initialize the windows using cvNamedWindow. I'm not sure about this, though.

Related

Glib: Calling a iterative loop function

Have a query on timeout calling and GMainContext. It is really confusing to me
Suppose I have the codes below (a bit incomplete, just for demonstration). I use normal Pthreads to create a thread. Within the thread, I run Glib functionality and created a GMainContext (stored within l_app.context).
I then created a source to run the function check_cmd iteratively at about 1 sec interval. This callback (or could we call it a thread?) will check for command from other threads( Pthreads not shown here for update in cmd status). From here onwards, there are two specific command
One to start a looping function
The other to end the looping function
I have done and thought of two ways to create the function and set them to run iteratively.
To create another timeout
using the same method of creating check_cmd
Essentially both to me are pretty much essentially the same method, when I tried both of them. Plan A (as I called it) does not work but Plan B ...actually run at least once. So I would like to know how to fix them...
Or maybe I should use g_source_add_child_source() instead?
In Summary, my question is
when you created a new context and push it to become the default context, do all subsequent function that require main_context will refer to this context?
in a nut shell, how do you add new sources when a loop is already running, ie like my cases
lastly, it is okay to quit the main loop within the callback you have created?
Here is my pseudocode
#include <glib.h>
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
#define PLAN_A 0
typedef struct
{
GMainContext *context;
GMainLoop *loop;
}_App;
static _App l_app;
guint gID;
gboolean
time_cycle(gpointer udata)
{
g_print("I AM THREADING");
return true;
}
gboolean
check_cmd_session(NULL )
{
while(alive) /// alive is a boolean value that is shared with other threads(not shown)
{
if(start)
{
/// PLAN A
//// which context does this add to ??
#if PLAN_A
g_timeout_add_seconds(10, (GSourceFunc)timeout, NULL);
#else
/// or should i use PLAN B
GSource* source = g_timeout_source_new(1000);
gID = g_source_set_callback(source,
(GSourceFunc)time_cycle,
NULL,
NULL);
g_source_attach(source, l_app.context);
#endif
}
else
{
#if PLAN_A
g_source_remove(gID);
#else
}
}
g_main_loop_quit (l_app.loop);
return FALSE;
}
void*
liveService(Info *info)
{
l_app.context = g_main_context_new ();
g_main_context_push_thread_default(l_app.context);
GSource* source = g_timeout_source_new(1000);
g_source_set_callback(source,
(GSourceFunc)check_cmd_session,
NULL,
NULL);
/// make it run
g_source_attach(source, l_app.context);
g_main_loop_run (l_app.loop);
pthread_exit(NULL);
}
int main()
{
pthread_t tid[2];
int thread_counter = 0;
err = pthread_create(&(tid[thread_counter]), NULL, &live, &info);
if (err != 0)
{
printf("\n can't create live thread :[%s]", strerror(err));
}
else
{
printf("--> Thread for Live created successfully\n");
thread_counter++;
}
/**** other threads are build not shown here */
for(int i = 0; i < 2; i++)
{
printf("Joining the %d threads \n", i);
pthread_join(tid[i],NULL);
}
return 0;
}
In Summary, my question is
when you created a new context and push it to become the default context, do all subsequent function that require main_context will
refer to this context?
Functions that are documented as using the thread-default main context will use the GMainContext which has been most recently pushed with g_main_context_push_thread_default().
Functions that are documented as using the global default main context will not. They will use the GMainContext which is created at init time and which is associated with the main thread.
g_timeout_add_seconds() is documented as using the global default main context. So you need to go with plan B if you want the timeout source to be attached to a specific GMainContext.
in a nut shell, how do you add new sources when a loop is already running, ie like my cases
g_source_attach() works when a main context is being iterated.
lastly, it is okay to quit the main loop within the callback you have created?
Yes, g_main_loop_quit() can be called at any point.
From your code, it looks like you’re not creating a new GMainLoop for each GMainContext and are instead assuming that one GMainLoop will somehow work with all GMainContexts in the process. That’s not correct. If you’re going to use GMainLoop, you need to create a new one for each GMainContext you create.
All other things aside, you might find it easier to use GLib’s threading functions rather than using pthread directly. GLib’s threading functions are portable to other platforms and a little bit easier to use. Given that you’re already linking to libglib, using them would cost nothing extra.

How to create monitoring thread?

I got a question while I'm doing for ray-tracing stuff.
I have created multiple threads to split whole image to be processed and let it process its allocated task. Threads work well as it is intended. I would like to monitor the work progress in real-time.
To resolve this problem, I have created one more thread to monitor current state.
Here is the monitoring pseudo-code:
/* Global var */
int cnt = 0; // count the number of row processed
void* render_disp(void* arg){ // thread for monitoring current render-processing
/* monitoring global variable and calculate percentage to display */
double result = 100.*cnt/(h-1);
fprintf(stderr,"\r3.2%f%% of image is processed!", result);
}
void* process(void* arg){ // multiple threads work here
// Rendering process
for(........)
pthread_mutex_lock(&lock);
cnt++;
pthread_mutex_unlock(&lock);
for(........)
}
I wrote the code for initialization of pthread and mutex in main() function.
Basically, I think this monitoring thread should display current state but this thread seems to be called only once and quit.
How do I change this code to this thread function to be called until the whole rendering is finished?

Display Opencv image send by event to Qt GUI

I am trying for days now to find a proper solution to the following problem(stated below), and I have no more ideas now. Therefore, I need help from more experienced devs:
I have a class independant from QT (and I want it to stay like this) that generate openCV images in a secundary thread. Then it raise an event which pass the images.
while (1)
{
if (timerActived & this->_camReady)
{
vector<Mat>* images = new vector<Mat>;
images = this->AcquireImg();
__raise this->frameAcquired(images);
}
this_thread::sleep_for(chrono::milliseconds(_frameTimeLaps));
}
This event is hooked by my MainWindow and a method is supposed to display my images on my GUI.
void MainWindow::displayFrame(vector<Mat>* frames) {
vector<Mat>* frames2 = new vector<Mat>();
frames2 = frames;
for (int i = 0; i < frames2->size(); ++i) {
this->camFrames->at(i)->showImage(frames2->at(i));
}
}
During the runTime, when the Main Thread access the images I get the following error :
ASSERT failure in QCoreApplication::sendEvent: "Cannot send events to objects owned by a different thread.
I understand that I am not using threads properly, but I have no idea how to create an asynchronous event now.
Thank you for help,
Valentin
EDIT 1 :
It seems to me that the real question is : can I use normal std::thread and native event to communicate with the main Qthread ?
Here is my solution to solve the problem, after all.
The idea is to give the main thread access to the Mat coming from the second thread. So I modified my function displayFrame like this :
void MainWindow::displayFrame(vector<Mat>* frames) {
//QThread::
QThread* this_thread = QThread::currentThread();
FrameWrapper* worker = new FrameWrapper();
worker->moveToThread(this_thread);
QObject::connect(worker, SIGNAL(frameSent(vector<Mat>*)), this, SLOT(showImg(vector<Mat>*)));
emit worker->frameSent(frames);
//this->camFrames->at(0)->showImage(frame);
}
The new function showImg is doing the job of the old displayFrame. The signal/slot process permit to access image from the GUI Thread as wanted.

Does Arduino support threading?

I have a couple of tasks to do with arduino but one of them takes very long time, so I was thinking to use threads to run them simultaneously.
I have an Arduino Mega
[Update]
Finally after four years I can install FreeRTOS in my arduino mega. Here is a link
In short: NO.
But you may give it a shot at:
http://www.kwartzlab.ca/2010/09/arduino-multi-threading-librar/
(Archived version: https://web.archive.org/web/20160505034337/http://www.kwartzlab.ca/2010/09/arduino-multi-threading-librar
Github: https://github.com/jlamothe/mthread
Not yet, but I always use this Library with big projects:
https://github.com/ivanseidel/ArduinoThread
I place the callback within a Timer interrupt, and voilá! You have pseudo-threads running on the Arduino...
Just to make this thread more complete: there are also protothreads which have very small memory footprint (couple bytes if I remember right) and preserve variables local to thread; very handy and time saving (far less finite state machines -> more readable code).
Examples and code:
arduino-class / ProtoThreads wiki
Just to let you know what results you may expect: serial communication # 153K6 baudrate with threads for: status diodes blinking, time keeping, requested functions evaluation, IO handling and logic and all on atmega328.
Not real threading but TimedActions are a good alternative for many uses
http://playground.arduino.cc/Code/TimedAction#Example
Of course, if one task blocks, the others will too, while threading can let one task freeze and the others will continue...
No you can't but you can use Timer interrupt.
Ref : https://www.teachmemicro.com/arduino-timer-interrupt-tutorial/
The previous answer is correct, however, the arduino generally runs pretty quick, so if you properly time your code, it can accomplish tasks more or less simultaneously.
The best practice is to make your own functions and avoid putting too much real code in the default void loop
You can use arduinos
It is designed for Arduino environment. Features:
Only static allocation (no malloc/new)
Support context switching when delaying execution
Implements semaphores
Lightweight, both cpu and memory
I use it when I need to receive new commands from bluetooth/network/serial while executing the old ones and the old ones have delay in them.
One thread is the sever thread that does the following loop:
while (1) {
while ((n = Serial.read()) != -1) {
// do something with n, like filling a buffer
if (command_was_received) {
arduinos_create(command_func, arg);
}
}
arduinos_yield(); // context switch to other threads
}
The other is the command thread that executes the command:
int command_func(void* arg) {
// move some servos
arduinos_delay(1000); // wait for them to move
// move some more servos
}
Arduino does not support multithread programming.
However there have been some workarounds, for example the one in this project (you can install it also from the Arduino IDE).
It seems you have to define the schedule time yourself while in a real multithread environment it is the OS that decides when to execute tasks.
Alternatively you can use protothreads
The straight answer is No No No!. There are some alternatives but you can't expect a perfect multi threading functionality from an arduino mega. You can use arduino due or lenado for multithreading like below-
void loop1(){
}
void loop2(){
}
void loop3(){
}
Normally, I handle those types of cases in backend. You can run the main code in a server while using Arduino to just collect inputs and show outputs. In such cases I would prefer nodemcu which has built in wifi.
Thread NO!
Concurrent YES!
You can run different tasks concurrently with FreeRTOS library.
https://www.arduino.cc/reference/en/libraries/freertos/
void TaskBlink( void *pvParameters );
void TaskAnalogRead( void *pvParameters );
// Now set up two tasks to run independently.
xTaskCreate(
TaskBlink
, (const portCHAR *)"Blink" // A name just for humans
, 128 // Stack size
, NULL
, 2 // priority
, NULL );
xTaskCreate(
TaskAnalogRead
, (const portCHAR *) "AnalogRead"
, 128 // This stack size can be checked & adjusted by reading Highwater
, NULL
, 1 // priority
, NULL );
void TaskBlink(void *pvParameters) // This is a task.
{
(void) pvParameters;
// initialize digital pin 13 as an output.
pinMode(13, OUTPUT);
for (;;) // A Task shall never return or exit.
{
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
vTaskDelay( 1000 / portTICK_PERIOD_MS ); // wait for one second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
vTaskDelay( 1000 / portTICK_PERIOD_MS ); // wait for one second
}
}
void TaskAnalogRead(void *pvParameters) // This is a task.
{
(void) pvParameters;
// initialize serial communication at 9600 bits per second:
Serial.begin(9600);
for (;;)
{
// read the input on analog pin 0:
int sensorValue = analogRead(A0);
// print out the value you read:
Serial.println(sensorValue);
vTaskDelay(1); // one tick delay (15ms) in between reads for stability
}
}
Just take care!
When different tasks tried to reach variables at the same time, like i2c communication line or sd card module. Use Semaphores and mutexes
https://www.geeksforgeeks.org/mutex-vs-semaphore/.
Arduino does not supports threading. However, you can do the next best thing and structure your code around state machines running in interleaving.
While there are lots of ways to implement your tasks as state machines, I recommend this library (https://github.com/Elidio/StateMachine). This library abstracts most of the process.
You can create a state machine as a class like this:
#include "StateMachine.h"
class STATEMACHINE(Blink) {
private:
int port;
int waitTime;
CREATE_STATE(low);
CREATE_STATE(high);
void low() {
digitalWrite(port, LOW);
*this << &STATE(high)<< waitTime;
}
void high() {
digitalWrite(port, HIGH);
*this << &STATE(low)<< waitTime;
}
public:
Blink(int port = 0, int waitTime = 0) :
port(port),
waitTime(waitTime),
INIT_STATE(low),
INIT_STATE(high)
{
pinMode(port, OUTPUT);
*this << &STATE(low);
}
};
The macro STATEMACHINE() abstracts the class inheritances, the macro CREATE_STATE() abstracts the state wrapper creation, the macro INIT_STATE() abstracts method wrapping and the macro STATE() abstracts state wrapper reference within the state machine class.
State transition is abstracted by << operator between the state machine class and the state, and if you want a delayed state transition, all you have to do is to use that operator with an integer, where the integer is the delay in millisseconds.
To use the state machine, first you have to instantiate it. Declaring an reference to the class in global space while instantiating it with new on setup function might do the trick
Blink *led1, *led2, *led3;
void setup() {
led1 = new Blink(12, 300);
led2 = new Blink(11, 500);
led3 = new Blink(10, 700);
}
Then you run the states on loop.
void loop() {
(*led2)();
(*led1)();
(*led3)();
}

Problem waking up multiple threads using condition variable API in win32

I have a problem in understanding how the winapi condition variables work.
On the more specific side, what I want is a couple of threads waiting on some condition. Then I want to use the WakeAllConditionVariable() call to wake up all the threads so that they can do work. Besides the fact that i just want the threads started, there isn't any other prerequisite for them to start working ( like you would have in an n producer / n consumer scenario ).
Here's the code so far:
#define MAX_THREADS 4
CONDITION_VARIABLE start_condition;
SRWLOCK cond_rwlock;
bool wake_all;
__int64 start_times[MAX_THREADS];
Main thread:
int main()
{
HANDLE h_threads[ MAX_THREADS ];
int tc;
for (tc = 0; tc < MAX_THREADS; tc++)
{
DWORD tid;
h_threads[tc] = CreateThread(NULL,0,(LPTHREAD_START_ROUTINE)thread_routine,(void*)tc,0,&tid);
if( h_threads[tc] == NULL )
{
cout << "Error while creating thread with index " << tc << endl;
continue;
}
}
InitializeSRWLock( &cond_rwlock );
InitializeConditionVariable( &start_condition );
AcquireSRWLockExclusive( &cond_rwlock );
// set the flag to true, then wake all threads
wake_all = true;
WakeAllConditionVariable( &start_condition );
ReleaseSRWLockExclusive( &cond_rwlock );
WaitForMultipleObjects( tc, h_threads, TRUE, INFINITE );
return 0;
}
And here is the code for the thread routine:
DWORD thread_routine( PVOID p_param )
{
int t_index = (int)(p_param);
AcquireSRWLockShared( &cond_rwlock );
// main thread sets wake_all to true and calls WakeAllConditionVariable()
// so this thread should start doing the work (?)
while ( !wake_all )
SleepConditionVariableSRW( &start_condition,&cond_rwlock, INFINITE,CONDITION_VARIABLE_LOCKMODE_SHARED );
QueryPerformanceCounter((LARGE_INTEGER*)&start_times[t_index]);
// do the actual thread related work here
return 0;
}
This code does not do what i would expect it to do. Sometimes just one thread finishes the job, sometimes two or three, but never all of them. The main function never gets past the WaitForMultipleObjects() call.
I'm not exactly sure what I've done wrong, but I would assume some synchronization issue somewhere ?
Any help would be appreciated. (sorry if I re-posted older topic with different dressing :)
You initialize the cond_rwlock and start_condition variables too late. Move the code up, before you start the threads. A thread is likely to start running right away, especially on a multi-core machine.
And test the return values of api functions. You don't know why it doesn't work because you never check for failure.

Resources