C++ thread and mutex and condition variable - multithreading

findsmallest common multiple of 10-million numbers in the queue does not exceed 10,000
I killed 2 days to sort out but I just do not understand! please help me
#include <condition_variable>
#include <mutex>
#include <thread>
#include <iostream>
#include <queue>
#include <chrono>
#include <cmath>
#include <map>
#include <cstdlib>
#include <fstream>
#include <ctime>
using namespace std;
int main()
{
std::map <int, int> NOK;
map<int, int> snok;
std::queue<int> oche;
std::mutex m;
std::condition_variable cond_var;
bool done = false;
bool notified = false;
std::thread filev([&]() {
//std::unique_lock<std::mutex> lock(m);
ifstream in; // Поток in будем использовать для чтения
int ch;
in.open("/home/akrasikov/prog/output.txt");
while(!in.eof()){
if (oche.size()>9999){
std::this_thread::sleep_for(std::chrono::milliseconds(3));
std::unique_lock<std::mutex> lock(m);
} else {
in>>ch;
oche.push(ch);
}
}
notified = true;
cond_var.notify_one();
done = true;
cond_var.notify_one();
});
std::thread nok([&]() {
std::unique_lock<std::mutex> lock(m);
while (!done) {
while (!notified) { // loop to avoid spurious wakeups
cond_var.wait(lock);
}
while (!oche.empty()) {
ch=oche.front();
oche.pop();
int j=2;
while (j < sqrt((double)ch)+1){
int s=0;
while(!(ch%j)){
s++;
ch/=j;
}
if (s > 0 && NOK[j] < s){
NOK[j] = s;
}
j++;
}
if (NOK[ch] == 0) NOK[ch]++;
}
long int su=1;
int temp=-1;
int step=0;
int sa=1;
std::cout << " NOK= ";
for (std::map<int, int>::iterator it=NOK.begin(); it!=NOK.end(); it++){
for (int i=0; i<it->second; i++){
su*=it->first;
sa=it->first;
if (temp<sa && sa >1){
temp=sa;
step=1;
} else {
if(sa>1)
step++;
}
}
cout<< temp << "^"<< step << " * " ;
}
std::cout << "su = " << su << '\n';
}
notified = false;
});
filev.join();
nok.join();
}
This program does not work! how come? what's wrong? it just starts and hangs, but if you do not delete is code
if (oche.size()>9999){
std::this_thread::sleep_for(std::chrono::milliseconds(3));
std::unique_lock<std::mutex> lock(m);
} else {
and
while (!done) {
while (!notified) { // loop to avoid spurious wakeups
cond_var.wait(lock);
}
everything works help plz

From what I understand of your problem, you have 3 problems
Conpute the least common multiple for a list of 1M elements
You want to have one thread that produces the elements and one that consumes it. They transfer it through a buffer (a queue in your case)
Your queue cannot exceed 10K elements
In my implementation I m generating the numbers randomly and using condition variables to coordinate between the threads.
Note that the LCM is associative so you can compute it recursively, not matter what the order is.
Here is the code but please DO NOT POST DIRTY CODE LIKE YOU DID NEXT TIME OR EVERYONE will kick you out.
Here is the code
#include <condition_variable>
#include <mutex>
#include <thread>
#include <iostream>
#include <queue>
#include <chrono>
#include <cmath>
#include <map>
#include <cstdlib>
#include <fstream>
#include <ctime>
#include <atomic>
#include <random>
using namespace std;
std::mutex mutRandom;//use for multithreading for random variables
int getNextRandom()
{
std::lock_guard<std::mutex> lock(mutRandom);
// C++11 Random number generator
std::mt19937 eng (time(NULL)); // Mersenne Twister generator with a different seed at each run
std::uniform_int_distribution<int> dist (1, 1000000);
return dist(eng);
}
//thread coordination
std::mutex mut;
std::queue<int> data_queue;
std::condition_variable data_cond;
std::atomic<int> nbData=0;
std::atomic<int> currLCM=1;//current LCM
const unsigned int nbMaxData=100000;
const unsigned int queueMaxSize=10000;
//Arithmetic function, nothing to do with threads
//greatest common divider
int gcd(int a, int b)
{
for (;;)
{
if (a == 0) return b;
b %= a;
if (b == 0) return a;
a %= b;
}
}
//least common multiple
int lcm(int a, int b)
{
int temp = gcd(a, b);
return temp ? (a / temp * b) : 0;
}
/// Thread related part
//for producing the data
void produceData()
{
while (nbData<nbMaxData)
{
std::unique_lock<std::mutex> lk(mut);
data_cond.wait(lk,[]{
return data_queue.size()<queueMaxSize;
});
cout<<nbData<<endl;
++nbData;
data_queue.push(getNextRandom());
data_cond.notify_one();
lk.unlock();
}
cout<<"Producer done \n";
}
//for consuming the data
void consumeData()
{
while (nbData<nbMaxData)
{
std::unique_lock<std::mutex> lk(mut);
data_cond.wait(lk,[]{
return !data_queue.empty();
});
int currData=data_queue.front();
data_queue.pop();
lk.unlock();
currLCM = lcm(currLCM,currData);
}
cout<<"Consumer done \n";
}
int main()
{
std::thread thProduce(&produceData);
std::thread thConsume(&consumeData);
thProduce.join();//to wait for the producing thread to finish before the program closes
thConsume.join();//same thing for the consuming one
return 0;
}
Hope that helps,

Related

Wait for thread queue to be empty

I am new to C++ and multithreading applications. I want to process a long list of data (potentially several thousands of entries) by dividing its entries among a few threads. I have retrieved a ThreadPool class and a Queue class from the web (it is my first time tackling the subject). I construct the threads and populate the queue in the following way (definitions at the end of the post):
ThreadPool *pool = new ThreadPool(8);
std::vector<std::function<void(int)>> *caller =
new std::vector<std::function<void(int)>>;
for (size_t i = 0; i < Nentries; ++i)
{
caller->push_back(
[=](int j){func(entries[i], j);});
pool->PushTask((*caller)[i]);
}
delete pool;
The problem is that only a number of entries equaling the number of created threads are processed, as if the program does not wait for the queue to be empty. Indeed, if I put
while (pool->GetWorkQueueLength()) {}
just before the pool destructor, the whole list is correctly processed. However, I am afraid I am consuming too many resources by using a while loop. Moreover, I have not found anyone doing anything like it, so I think this is the wrong approach and the classes I use have some error. Can anyone find the error (if present) or suggest another solution?
Here are the classes I use. I suppose the problem is in the implementation of the destructor, but I am not sure.
SynchronizeQueue.hh
#ifndef SYNCQUEUE_H
#define SYNCQUEUE_H
#include <list>
#include <mutex>
#include <condition_variable>
template<typename T>
class SynchronizedQueue
{
public:
SynchronizedQueue();
void Put(T const & data);
T Get();
size_t Size();
private:
SynchronizedQueue(SynchronizedQueue const &)=delete;
SynchronizedQueue & operator=(SynchronizedQueue const &)=delete;
std::list<T> queue;
std::mutex mut;
std::condition_variable condvar;
};
template<typename T>
SynchronizedQueue<T>::SynchronizedQueue()
{}
template<typename T>
void SynchronizedQueue<T>::Put(T const & data)
{
std::unique_lock<std::mutex> lck(mut);
queue.push_back(data);
condvar.notify_one();
}
template<typename T>
T SynchronizedQueue<T>::Get()
{
std::unique_lock<std::mutex> lck(mut);
while (queue.empty())
{
condvar.wait(lck);
}
T result = queue.front();
queue.pop_front();
return result;
}
template<typename T>
size_t SynchronizedQueue<T>::Size()
{
std::unique_lock<std::mutex> lck(mut);
return queue.size();
}
#endif
ThreadPool.hh
#ifndef THREADPOOL_H
#define THREADPOOL_H
#include "SynchronizedQueue.hh"
#include <atomic>
#include <functional>
#include <mutex>
#include <thread>
#include <vector>
class ThreadPool
{
public:
ThreadPool(int nThreads = 0);
virtual ~ThreadPool();
void PushTask(std::function<void(int)> func);
size_t GetWorkQueueLength();
private:
void WorkerThread(int i);
std::atomic<bool> done;
unsigned int threadCount;
SynchronizedQueue<std::function<void(int)>> workQueue;
std::vector<std::thread> threads;
};
#endif
ThreadPool.cc
#include "ThreadPool.hh"
#include "SynchronizedQueue.hh"
void doNothing(int i)
{}
ThreadPool::ThreadPool(int nThreads)
: done(false)
{
if (nThreads <= 0)
{
threadCount = std::thread::hardware_concurrency();
}
else
{
threadCount = nThreads;
}
for (unsigned int i = 0; i < threadCount; ++i)
{
threads.push_back(std::thread(&ThreadPool::WorkerThread, this, i));
}
}
ThreadPool::~ThreadPool()
{
done = true;
for (unsigned int i = 0; i < threadCount; ++i)
{
PushTask(&doNothing);
}
for (auto& th : threads)
{
if (th.joinable())
{
th.join();
}
}
}
void ThreadPool::PushTask(std::function<void(int)> func)
{
workQueue.Put(func);
}
void ThreadPool::WorkerThread(int i)
{
while (!done)
{
workQueue.Get()(i);
}
}
size_t ThreadPool::GetWorkQueueLength()
{
return workQueue.Size();
}
You can push tasks saying "done" instead of setting "done" via atomic variable.
So that each thread will exit by itself when seeing "done" task, and no earlier. In destructor you only need to push these tasks and join threads. This is called "poison pill".
Alternatively, if you insist on your current design with done variable, you can wait on the same condition you already have:
std::unique_lock<std::mutex> lck(mut);
while (!queue.empty())
{
condvar.wait(lck);
}
But then you'll need to change your notify_one to notify_all, and this may be sub-optimal.
I want to process a long list of data (potentially several thousands of entries) by dividing its entries among a few threads.
You can do that with parallel algorithms, like tbb::parallel_for:
#include <tbb/parallel_for.h>
#include <vector>
void func(int entry);
int main () {
std::vector<int> entries(1000000);
tbb::parallel_for(size_t{0}, entries.size(), [&](size_t i) { func(entries[i]); });
}
If you need sequential thread ids, you can do:
void func(int element, int thread_id);
template<class C>
inline auto make_range(C& c) -> decltype(tbb::blocked_range<decltype(c.begin())>(c.begin(), c.end())) {
return tbb::blocked_range<decltype(c.begin())>(c.begin(), c.end());
}
int main () {
std::vector<int> entries(1000000);
std::atomic<int> thread_counter{0};
tbb::parallel_for(make_range(entries), [&](auto sub_range) {
static thread_local int const thread_id = thread_counter.fetch_add(1, std::memory_order_relaxed);
for(auto& element : sub_range)
func(element, thread_id);
});
}
Alternatively, there is std::this_thread::get_id.

Using CLOCK_MONOTONIC type in the 'condition variable' wait_for() notify() mechanism

I am using code that runs on ARM (not Intel processor). Running c++11 code example (CODE A) from: http://www.cplusplus.com/reference/condition_variable/condition_variable/wait_for/ to test the wait_for() mechanism. This is not working right - looks like the wait_for() does not wait. In Intel works fine. After some research and using pthread library directly and setting MONOTONIC_CLOCK definition, solves the issue (CODE B).
(Running on ARM is not the issue)
My problem is :
How can I force the C++11 API wait_for() to work with MONOTONIC_CLOCK?
Actually I would like to stay with 'CODE A' but with the support or setting of MONOTONIC_CLOCK.
Thanks
CODE A
// condition_variable::wait_for example
#include <iostream> // std::cout
#include <thread> // std::thread
#include <chrono> // std::chrono::seconds
#include <mutex> // std::mutex, std::unique_lock
#include <condition_variable> // std::condition_variable, std::cv_status
std::condition_variable cv;
int value;
void read_value() {
std::cin >> value;
cv.notify_one();
}
int main ()
{
std::cout << "Please, enter an integer (I'll be printing dots): \n";
std::thread th (read_value);
std::mutex mtx;
std::unique_lock<std::mutex> lck(mtx);
while
(cv.wait_for(lck,std::chrono::seconds(1))==std::cv_status::timeout)
{
std::cout << '.' << std::endl;
}
std::cout << "You entered: " << value << '\n';
th.join();
return 0;
}
CODE B
#include <sys/time.h>
#include <unistd.h>
#include <iostream> // std::cout
#include <thread> // std::thread
#include <chrono> // std::chrono::seconds
#include <mutex> // std::mutex, std::unique_lock
#include <condition_variable> // std::condition_variable, std::cv_status
const size_t NUMTHREADS = 1;
pthread_mutex_t mutex;
pthread_cond_t cond;
int value;
bool done = false;
void* read_value( void* id )
{
const int myid = (long)id; // force the pointer to be a 64bit integer
std::cin >> value;
done = true;
printf( "[thread %d] done is now %d. Signalling cond.\n", myid, done
);
pthread_cond_signal( &cond );
}
int main ()
{
struct timeval now;
pthread_mutexattr_t Attr;
pthread_mutexattr_init(&Attr);
pthread_mutexattr_settype(&Attr, PTHREAD_MUTEX_RECURSIVE);
pthread_mutex_init(&mutex, &Attr);
pthread_condattr_t CaAttr;
pthread_condattr_init(&CaAttr);
pthread_condattr_setclock(&CaAttr, CLOCK_MONOTONIC);
pthread_cond_init(&cond, &CaAttr);
std::cout << "Please, enter an integer:\n";
pthread_t threads[NUMTHREADS];
int t = 0;
pthread_create( &threads[t], NULL, read_value, (void*)(long)t );
struct timespec ts;
pthread_mutex_lock( &mutex );
int rt = 0;
while( !done )
{
clock_gettime(CLOCK_MONOTONIC, &ts);
ts.tv_sec += 1;
rt = pthread_cond_timedwait( & cond, & mutex, &ts );
std::cout << "..." << std::endl;
}
pthread_mutex_unlock( & mutex );
std::cout << "You entered: " << value << '\n';
return 0;
}
The documentation for std::condition_variable::wait_for says:
A steady clock is used to measure the duration.
std::chrono::steady_clock:
Class std::chrono::steady_clock represents a monotonic clock. The time points of this clock cannot decrease as physical time moves forward.
Unfortunately, this is gcc Bug 41861 (DR887) - (DR 887)(C++0x) does not use monotonic_clock that it uses system_clock instead of steady_clock for condition variables.
One solution is to use wait_until (be sure to read Notes section) function that allows to specify durations relative to a specific clock. E.g.:
cv.wait_until(lck, std::chrono::steady_clock::now() + std::chrono::seconds(1))

C++11: Thread pool for successive input reduction

Consider a container of type T elements, e.g. std::vector<T> input, and an excessively time-consuming function T f(const T&, const T&). I wish to apply f on input until one single element remains - in parallel.
I am new to C++ so I adapted an existing implementation for a Thread pool pattern in C++11 to my needs, in particular this example.
Code for discussion:
#include <thread>
#include <iostream>
#include <algorithm>
#include <vector>
#include <queue>
#include <memory>
#include <future>
#include <functional>
#include <stdexcept>
#include "ThreadPool.h"
template<class T>
T ParallelReduction(ThreadPool& pool, const std::vector<T>& input, const std::function<T(T,T)>& func)
{
std::queue<std::future<T>> futureResults;
std::for_each(input.begin(), input.end(), [&futureResults](T e)
{
// T -> future<T>
futureResults.push(std::async([e]{return e;}));
});
for (;;)
{
if (futureResults.size() > 1)
{
// Get two "old" partial results, forward them to func(T,T)
T a = futureResults.front().get();
futureResults.pop();
T b = futureResults.front().get();
futureResults.pop();
futureResults.push(pool.enqueue([a, b, func]{return func(a,b);}));
}
else
{
// all input and partial result elements have been processed once, return the final result
return futureResults.front().get();
}
}
}
int main(int argc, char *argv[])
{
ThreadPool pool(4);
std::vector<int> input(12);
int n = 0;
std::generate(input.begin(), input.end(), [&n]{ return ++n; });
auto sum = ParallelReduction<int>(pool, input, [](int a, int b){return a + b;});
std::cout << sum << "\n";
auto fibonacci = ParallelReduction<int>(pool, input, [](int a, int b){return a * b;});
std::cout << fibonacci << "\n";
return 0;
}
My questions:
Is it possible to implement this without introducing futureResults, i.e. a solution that works "inline" on std::vector<T>& input?
If not, can we optimize the way we assign values T to std::future<T> in the first part?
Is it more efficient to forward std::future<T> to our function func instead of T, i.e. let the child thread wait and access the value of a and b and not the main thread?
The parts inside the loop look awful - any ideas?
Where can I optimize the code?

error C4716: function : must return a value

So I am trying to use pthread libraries for Visual C++(2012) and I get this error error C4716: 'print_message' : must return a value
Here's the code
#include "stdafx.h"
#include <iostream>
#include "pthread.h"
using namespace std;
void* print_message(void *)
{
cout << "Threading\n";
}
int main()
{
pthread_t t1;
pthread_create(&t1, NULL, print_message, NULL);
cout << "Hello";
void* result;
pthread_join(t1,&result);
return 0;
}
Add return NULL; to print_message. I'll bet you need to name the argument too.

Multithread Directory and File Search

I am new to semaphores and the concepts of mutual exclusion. I am supposed to recursively text search in files through directories using multithreading. The number of threads is to be given by the user.
The issue with this code is it goes through one directory and then waits. I cannot figure out what is wrong.I am getting a segmentation fault error. Cannot figure out why is this happening.
#include <iostream>
#include <sys/wait.h>
#include <sys/types.h>
#include <pthread.h>
#include <string.h>
#include <unistd.h>
#include <dirent.h>
#include <sys/stat.h>
#include <fstream>
#include <limits.h>
#include <stdlib.h>
#include <semaphore.h>
using namespace std;
#include <stdio.h>
int iDirectories=0;
pthread_mutex_t mutex=PTHREAD_MUTEX_INITIALIZER;
sem_t semaphore1;
char searchStringThread[PATH_MAX];
int directories=0;
class directoryQueue
{
private:
struct Node
{
char directoryPath[PATH_MAX];
Node *next;
};
Node *front;
Node *rear;
Node *nodeCount;
public:
directoryQueue(void)
{
front=NULL;
rear=NULL;
nodeCount=0;
}
void Enqueue(char array[PATH_MAX])
{
Node *newNode;
newNode=new Node;
strcpy(newNode->directoryPath,array);
newNode->next=NULL;
if(isEmpty())
{
front=newNode;
rear=newNode;
}
else
{
rear->next=newNode;
rear=newNode;
}
nodeCount++;
}
char * Dequeue(void)
{
Node *temp;
if (isEmpty())
cout << "Error ! Empty Queue "<<endl;
else
{
char *deque;
deque=new char[PATH_MAX];
strcpy(deque,front->directoryPath);
temp = front->next;
front = temp;
nodeCount--;
return deque;
}
}
bool isEmpty(void)
{
if(nodeCount)
return false;
else
return true;
}
void makeNull(void)
{
while(!isEmpty())
{
Dequeue();
}
}
~directoryQueue(void)
{
makeNull();
}
};
directoryQueue saveDirectory;
void *threadHandler(void *)
{
int thpath_length;
char thPath[PATH_MAX];
char saveITDirectory[PATH_MAX];
char itDirectory[PATH_MAX];
int threadCount;
struct dirent *iWalker;
DIR *iDirectory;
pthread_mutex_lock(&mutex);
threadCount=iDirectories++;
pthread_mutex_unlock(&mutex);
sem_wait(&semaphore1);
pthread_mutex_lock(&mutex);
strcpy(itDirectory,saveDirectory.Dequeue());
pthread_mutex_unlock(&mutex);
iDirectory=opendir(itDirectory);
if(iDirectory==NULL)
{
cout<<"Error"<<endl;
cout<<itDirectory<<" Cannot be Opened"<<endl;
exit(10000);
}
while((iWalker=readdir(iDirectory)) !=NULL)
{
if(iWalker->d_type==DT_REG)
{
strcpy(saveITDirectory,iWalker->d_name);
cout<<itDirectory<<"/"<<endl;
if (strcmp (saveITDirectory, "..") == 0 ||
strcmp (saveITDirectory, ".") == 0)
{
continue;
}
else
{
thpath_length = snprintf(thPath,PATH_MAX,"%s/%s",itDirectory,saveITDirectory);
cout<<thPath<<endl;
if (thpath_length >= PATH_MAX)
{
cout<<"Path is too long"<<endl;
exit (1000);
}
ifstream openFile;
openFile.open(thPath);
char line[1500];
int currentLine = 0;
if (openFile.is_open()) {
while (openFile.good()) {
currentLine++;
openFile.getline(line, 1500);
if (strstr(line, searchStringThread) != NULL){
cout<<thPath<<": "<<currentLine<<": "<<line<<endl;
cout<<"This was performed by Thread no. "<<threadCount<<endl;
cout<<"ID :"<<pthread_self();
}
}
}
openFile.close();
}
}
if (closedir (iDirectory))
{
cout<<"Unable to close "<<itDirectory<<endl;
exit (1000);
}
}
}
void walkThroughDirectory(char directory_name[PATH_MAX],char searchString[PATH_MAX])
{
DIR * directory;
struct dirent * walker;
char d_name[PATH_MAX];
int path_length;
char path[PATH_MAX];
directory=opendir(directory_name);
if(directory==NULL)
{
cout<<"Error"<<endl;
cout<<directory_name<<" Cannot be Opened"<<endl;
exit(10000);
}
while((walker=readdir(directory)) !=NULL)
{
strcpy(d_name,walker->d_name);
cout<<directory_name<<"/"<<endl;
if (strcmp (d_name, "..") == 0 ||
strcmp (d_name, ".") == 0)
{
continue;
}
else
{
path_length = snprintf(path,PATH_MAX,"%s/%s",directory_name,d_name);
cout<<path<<endl;
if (path_length >= PATH_MAX)
{
cout<<"Path is too long"<<endl;
exit (1000);
}
if(walker->d_type==DT_DIR)
{
pthread_mutex_lock(&mutex);
saveDirectory.Enqueue(path);
pthread_mutex_lock(&mutex);
sem_post(&semaphore1);
directories++;
walkThroughDirectory (path,searchString);
}
else if(walker->d_type==DT_REG)
{
ifstream openFile;
openFile.open(path);
char line[1500];
int currentLine = 0;
if (openFile.is_open()) {
while (openFile.good()) {
currentLine++;
openFile.getline(line, 1500);
if (strstr(line, searchString) != NULL)
cout<<path<<": "<<currentLine<<": "<<line<<endl;
}
}
openFile.close();
}
}
}
if (closedir (directory))
{
cout<<"Unable to close "<<directory_name<<endl;
exit (1000);
}
}
int main(int argc,char *argv[])
{
char * name;
cout<<"Total Directories "<< directories<<endl;
name=get_current_dir_name();
cout<<"Current Directory is: "<<name<<endl;
sem_init(&semaphore1,0,0);
strcpy(searchStringThread,argv[1]);
int number_of_threads=atoi(argv[3]);
pthread_t threads[number_of_threads];
walkThroughDirectory(argv[2],argv[1]);
pthread_mutex_lock(&mutex);
saveDirectory.Enqueue(argv[2]);
pthread_mutex_unlock(&mutex);
sem_post(&semaphore1);
for(int i=0;i<number_of_threads;i++)
{
pthread_create(&threads[i],NULL,threadHandler,NULL);
}
for(int j=0;j<number_of_threads;j++)
{
pthread_join(threads[j],NULL);
}
while(saveDirectory.isEmpty())
{
cout<<"Queue is Empty"<<endl;
cout<<"Exiting"<<endl;
exit(10000);
}
free(name);
cout<<"Total Directories "<< directories<<endl;
return 0;
}
There's a simple bug where you lock a mutex twice instead of unlocking it when you're done:
pthread_mutex_lock(&mutex);
saveDirectory.Enqueue(path);
pthread_mutex_lock(&mutex);
should be:
pthread_mutex_lock(&mutex);
saveDirectory.Enqueue(path);
pthread_mutex_unlock(&mutex);
Note: this isn't to say that there aren't other problems - just that this is probably your immediate problem.
The biggest problem is that it looks like you put directories on the saveDirectory queue (so another thread can pull it off to work on it), then go ahead an process that directory recursively in the thread that just put it on the queue. I think you'll need to give some more thought on how the work will be divided among the threads.
A couple of more minor comments:
you might want to consider using std::string if that's permitted. It should make some of your string handling simpler (you leak memory from the data returned from directoryQueue::Dequeue(), for example)
if the primary reason for the existence of the directoryQueue class is to hold work items for multiple threads, then maybe it should manage it's own mutex so callers don't need to deal with that complexity

Resources