How to print the progress of cython's prange? - multithreading

Question

I'm using prange for parallelizing a loop in my cython code. As this loop takes a long time, I want to print its progress as it goes. A progress bar would be nice but anything that shows the progress would do. I tried to add some form of logging in the following fashion:
for index in prange(n, nogil = True, num_threads = 5):
if index == (index//1000)*1000:
printf("%d percent done", index*100/n)
This should print the progress whenever index % 1000 == 0.
The output of this is kind of random.
0 percent done
80 percent done
20 percent done
100 percent done
60 percent done
I assume that this is because prange does not assign threads to indexes starting from 0 and going up.
Is there any way to implement such a thing in Cython?
Thanks!

Answer 1

There are probably ways in Cython to achieve what you want, however I find it easier to use OpenMP from C/C++ and to call the functionality from Cython (thanks to C-verbatim-code since Cython 0.28 it is quite convenient).
One thing is clear, to achieve your goal you need some kind of synchronization between threads, which might impact performance. My strategy is simple: every thread reports "done", all done task are registered and they number is reported from time to time. The count of reported tasks is protected with a mutex/lock:
%%cython -c=/openmp --link-args=/openmp
cdef extern from * nogil:
r"""
#include <omp.h>
#include <stdio.h>
static omp_lock_t cnt_lock;
static int cnt = 0;
void reset(){
omp_init_lock(&cnt_lock);
cnt = 0;
}
void destroy(){
omp_destroy_lock(&cnt_lock);
}
void report(int mod){
omp_set_lock(&cnt_lock);
// start protected code:
cnt++;
if(cnt%mod == 0){
printf("done: %d\n", cnt);
}
// end protected code block
omp_unset_lock(&cnt_lock);
}
"""
void reset()
void destroy()
void report(int mod)
from cython.parallel import prange
def prange_with_report(int n):
reset() # reset counter and init lock
cdef int index
for index in prange(n, nogil = True, num_threads = 5):
report(n//10)
destroy() # release lock
Now calling: prange_with_report(100) would print done: 10\n, done: 20\n, ..., done: 100\n.
As a slight optimization, report could be called not for every index but only for example for index%100==0 - there will be less impact on the performance but also the reporting will be less precise.