Enforcing progress thread when overlapping communication and computation - multithreading

Question

This question is originating from here: Overlapping communication and computation taking 2.1 times as much time.
I've implemented Cannon's algorithm which performs distributed memory tensor-matrix multiplication. During this, I thought it would be clever to hide communication latencies by overlapping computation and communication.
Now I started micro-benchmarking the components i.e. the communication, the computation and the overlapped communication and computation, and something funny has come out of it. The overlapped operation is taking 2.1 times as long as the longest time taking operation of the two. Only sending a single message took 521639 us, only computing on the (same sized) data took 340435 us but the act of overlapping them took 1111500 us.
After numerous test-runs involving independent data buffers and valuable inputs in the form of comments, I have come to the conclusion that the problem is being caused by MPI's weird concept of progress.
The following is the desired behaviour:
only the thread identified by COMM_THREAD handles the communication and
all the other threads perform the computation.
If the above behaviour can be forced, in the above example, I expect to see the overlapped operation take ~521639 us.
Information:
The MPI implementation is by Intel as part of OneAPI v2021.6.0.
A single compute node has 2 sockets of Intel Xeon Platinum 8168 (2x 24 = 48 cores).
SMT is not being made use of i.e. each thread is pinned to a single core.
Data is being initialized before each experiment by being mapped to the corresponding memory nodes as required by the computation.
Benchmarking runs are preceded by 10 warm-up runs.
In the given example, the tensor is sized N=600 i.e. has 600^3 data-points. However, the same behaviour was observed for smaller sizes as well.
What I've tried:
Just making asynchronous calls in the overlap:
// ...
#define COMM_THREAD 0
// ...
#pragma omp parallel
{
if (omp_get_thread_num() == COMM_THREAD)
{
// perform the comms.
auto requests = std::array<MPI_Request, 4>{};
const auto r1 = MPI_Irecv(tens_recv_buffer_->data(), 2 * tens_recv_buffer_->size(), MPI_DOUBLE, src_proc_id_tens_,
2, MPI_COMM_WORLD, &requests[0]);
const auto s1 = MPI_Isend(tens_send_buffer_->data(), 2 * tens_send_buffer_->size(), MPI_DOUBLE,
target_proc_id_tens_, 2, MPI_COMM_WORLD, &requests[1]);
const auto r2 = MPI_Irecv(mat_recv_buffer_->data(), 2 * mat_recv_buffer_->size(), MPI_DOUBLE, src_proc_id_mat_,
3, MPI_COMM_WORLD, &requests[2]);
const auto s2 = MPI_Isend(mat_send_buffer_->data(), 2 * mat_send_buffer_->size(), MPI_DOUBLE,
target_proc_id_mat_, 3, MPI_COMM_WORLD, &requests[3]);
if (MPI_SUCCESS != s1 || MPI_SUCCESS != r1 || MPI_SUCCESS != s2 || MPI_SUCCESS != r2)
{
throw std::runtime_error("tensor_matrix_mult_mpi_sendrecv_error");
}
if (MPI_SUCCESS != MPI_Waitall(requests.size(), requests.data(), MPI_STATUSES_IGNORE))
{
throw std::runtime_error("tensor_matrix_mult_mpi_waitall_error");
}
}
else
{
const auto work_indices = schedule_thread_work(tens_recv_buffer_->get_n1(), 1);
shared_mem::tensor_matrix_mult(*tens_send_buffer_, *mat_send_buffer_, *result_, work_indices);
}
}
Trying manual progression:
if (omp_get_thread_num() == COMM_THREAD)
{
// perform the comms.
auto requests = std::array<MPI_Request, 4>{};
const auto r1 = MPI_Irecv(tens_recv_buffer_->data(), 2 * tens_recv_buffer_->size(), MPI_DOUBLE, src_proc_id_tens_,
2, MPI_COMM_WORLD, &requests[0]);
const auto s1 = MPI_Isend(tens_send_buffer_->data(), 2 * tens_send_buffer_->size(), MPI_DOUBLE,
target_proc_id_tens_, 2, MPI_COMM_WORLD, &requests[1]);
const auto r2 = MPI_Irecv(mat_recv_buffer_->data(), 2 * mat_recv_buffer_->size(), MPI_DOUBLE, src_proc_id_mat_,
3, MPI_COMM_WORLD, &requests[2]);
const auto s2 = MPI_Isend(mat_send_buffer_->data(), 2 * mat_send_buffer_->size(), MPI_DOUBLE,
target_proc_id_mat_, 3, MPI_COMM_WORLD, &requests[3]);
if (MPI_SUCCESS != s1 || MPI_SUCCESS != r1 || MPI_SUCCESS != s2 || MPI_SUCCESS != r2)
{
throw std::runtime_error("tensor_matrix_mult_mpi_sendrecv_error");
}
// custom wait-all to ensure COMM_THREAD makes progress happen
auto comm_done = std::array<int, 4>{0, 0, 0, 0};
auto all_comm_done = false;
while(!all_comm_done)
{
auto open_comms = 0;
for (auto request_index = std::size_t{}; request_index < requests.size(); ++request_index)
{
if (comm_done[request_index])
{
continue;
}
MPI_Test(&requests[request_index], &comm_done[request_index], MPI_STATUS_IGNORE);
++open_comms;
}
all_comm_done = open_comms == 0;
}
}
else
{
const auto work_indices = schedule_thread_work(tens_recv_buffer_->get_n1(), 1);
shared_mem::tensor_matrix_mult(*tens_send_buffer_, *mat_send_buffer_, *result_, work_indices);
}
Using the environment variables mentioned here: https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/environment-variables-for-async-progress-control.html in my job-script:
export I_MPI_ASYNC_PROGRESS=1 I_MPI_ASYNC_PROGRESS_THREADS=1 I_MPI_ASYNC_PROGRESS_PIN="0"
and then running the code in variant 1.
All of the above attempts have resulted in the same undesirable behaviour.
Question: How can I force only COMM_THREAD to participate in MPI progression?
Any thoughts, suggestions, speculations and ideas will be greatly appreciated. Thanks in advance.
Note: although the buffers tens_send_buffer_ and mat_send_buffer_ are accessed concurrently during the overlap, this is read-only access.