Pthread Scheduling policy and priority - multithreading

Question

I have four threads which are waiting on a condition variable and fifth thread posts condition variable when all four threads are waiting. When I set thread priority to maximum that is 99, threads switch takes a lot of time which is far from acceptable. Can anybody please take a look and tell what's happening ?
#define N_WORK_THREADS 4
pthread_mutex_t count_mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t condition_var = PTHREAD_COND_INITIALIZER;
void *functionCount1(void * arg);
void *functionCount2(void * arg);
int count = 0;
int valid = 0;
int thread_personal[N_WORK_THREADS];
static int display_thread_sched_attr(int id)
{
int policy, s;
struct sched_param param;
s = pthread_getschedparam(pthread_self(), &policy, &param);
if (s != 0) { printf("pthread_getschedparam"); return 1; }
printf("Thread Id=%d policy=%s, priority=%d\n",id,
(policy == SCHED_FIFO) ? "SCHED_FIFO" : (policy == SCHED_RR) ? "SCHED_RR" : (policy == SCHED_OTHER) ? "SCHED_OTHER" : "???",
param.sched_priority);
return 0;
}
int main(void)
{
pthread_t thread_work[N_WORK_THREADS];
pthread_t thread;
int i,s;
pthread_attr_t attr;
struct sched_param param;
s = pthread_attr_init(&attr);
if (s != 0) { printf("pthread_attr_init"); return 1; }
s = pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);
if (s != 0) { printf("pthread_attr_setinheritsched"); return 1; }
s = pthread_attr_setschedpolicy(&attr, SCHED_RR);
if (s != 0) { printf("pthread_attr_setschedpolicy"); return 1; }
param.sched_priority = 99;
s = pthread_attr_setschedparam(&attr, &param);
if (s != 0) { printf("pthread_attr_setschedparam"); return 1; }
for (i=0; i<N_WORK_THREADS; i++) { thread_personal[i] = 0; }
for (i=0; i<N_WORK_THREADS; i++) { pthread_create( &thread_work[i], &attr, &functionCount1, (void *)i); }
param.sched_priority = 99;
s = pthread_attr_setschedparam(&attr, &param);
if (s != 0) { printf("pthread_attr_setschedparam"); return 1; }
pthread_create( &thread, &attr, &functionCount2, (void *)N_WORK_THREADS);
for (i=0; i<N_WORK_THREADS; i++) { pthread_join( thread_work[i], NULL); }
pthread_join( thread, NULL);
for (i=0; i<N_WORK_THREADS; i++) { printf("Thread Id=%d Mutex USed=%d\n",i,thread_personal[i]); }
exit(EXIT_SUCCESS);
}
void *functionCount1(void * arg)
{
int i;
int id = (int) arg;
display_thread_sched_attr(id);
for(i=0; i<10; i++)
{
pthread_mutex_lock( &count_mutex );
thread_personal[id] += 1;
while (((count>>id) & 0x1) == 0)
{
pthread_cond_wait( &condition_var, &count_mutex );
}
count = count^ (1<<id);
printf("Thread Id %d: Valid = %d\n",id,valid);
pthread_mutex_unlock( &count_mutex );
}
return NULL;
}
void *functionCount2(void * arg)
{
int check;
int id = (int) arg;
display_thread_sched_attr(id);
check =0;
while (check < 10)
{
pthread_mutex_lock( &count_mutex );
if (count == 0)
{
pthread_cond_broadcast ( &condition_var );
count =0xF;
printf("Thread Id %d: Counter = %d\n",id,check);
valid = check++;
}
pthread_mutex_unlock( &count_mutex );
}
return NULL;
}

Answer 1

I'm unable to test your program with the scheduling policy code enabled because the program simply doesn't work when that's in there (as I mention in a comment: Linux 3.16.0 x86_64 with gcc 4.8.4).
But I'm guessing that your problem might be due to the loop in functionCount2():
while (check < 10)
{
pthread_mutex_lock( &count_mutex );
if (count == 0)
{
pthread_cond_broadcast ( &condition_var );
count =0xF;
printf("Thread Id %d: Counter = %d\n",id,check);
valid = check++;
}
pthread_mutex_unlock( &count_mutex );
}
In general, acquisition of mutex objects in pthreads is not guaranteed to be fair or FIFO (though to be honest, I'm not sure how thread scheduling policies might affect it). What I believe is happening is that this loop releases count_mutex then immediately re-acquires it even though other threads are blocked waiting to claim the mutex. And with the scheduling policy in place, this may occur until the thread uses its quantum.