mpi_run on multicore architecture --bind-to l3 or --bind-to core - multithreading

Question

I am running a code on a 24c architecture and would like to use one mpi rank for each set of three cores bound to a L3 cache bloc. So, 8 mpi ranks per socket, 16 per node, with 3 threads per rank. I think the following command line should apply
mpirun --bind-to l3 -np 16 gmx_mpi mdrun -nt 3
--bind-to binding the mpi ranks to each bloc of L3 cache, -np allocating 16 mpi ranks per node and a -nt a number of threads per MPI rank of 3. Is this the correct approach ?
If the core is capable of multithreading (2 threads) is it right to write
mpirun --bind-to l3 -np 16 gmx_mpi mdrun -nt 6
--bind-to core is I assume binding one MPI rank per core, with no spanning into threads, or spanning into 2 threads per core for exploiting MT, e.g.
mpirun --bind-to core -np 48 gmx_mpi mdrun -nt 2
with 48 ranks one per core on a 2-socket platform and 2 threads per core (MT)
Would you confirm ?

Answer 1

I always use this piece of code, that I inherited from somewhere many years ago, to print out bindings at runtime. For example, on my 4-core laptop:
dsh#e7390dh:binding$ mpicc -o bind bind.c utilities.c
dsh#e7390dh:binding$ mpirun -n 4 ./bind
Rank 2 on core 2,6 of node <e7390dh>
Rank 3 on core 3,7 of node <e7390dh>
Rank 0 on core 0,4 of node <e7390dh>
Rank 1 on core 1,5 of node <e7390dh>
i.e. each process is bound to one physical core but can run on either hypercore. If there is no binding you get a range, e.g. "on core [0-7]".
Hope this is useful.
bind.c:
#include <stdio.h>
#include <mpi.h>
void printlocation();
int main(void)
{
MPI_Init(NULL,NULL);
printlocation();
MPI_Finalize();
return 0;
}
utilities.c:
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <sched.h>
#include <mpi.h>
/* Borrowed from util-linux-2.13-pre7/schedutils/taskset.c */
static char *cpuset_to_cstr(cpu_set_t *mask, char *str)
{
char *ptr = str;
int i, j, entry_made = 0;
for (i = 0; i < CPU_SETSIZE; i++) {
if (CPU_ISSET(i, mask)) {
int run = 0;
entry_made = 1;
for (j = i + 1; j < CPU_SETSIZE; j++) {
if (CPU_ISSET(j, mask)) run++;
else break;
}
if (!run)
sprintf(ptr, "%d,", i);
else if (run == 1) {
sprintf(ptr, "%d,%d,", i, i + 1);
i++;
} else {
sprintf(ptr, "%d-%d,", i, i + run);
i += run;
}
while (*ptr != 0) ptr++;
}
}
ptr -= entry_made;
*ptr = 0;
return(str);
}
void printlocation()
{
int rank, namelen;
char hnbuf[MPI_MAX_PROCESSOR_NAME];
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
memset(hnbuf, 0, sizeof(hnbuf));
MPI_Get_processor_name(hnbuf, &namelen);
cpu_set_t coremask;
char clbuf[7 * CPU_SETSIZE];
memset(clbuf, 0, sizeof(clbuf));
(void)sched_getaffinity(0, sizeof(coremask), &coremask);
cpuset_to_cstr(&coremask, clbuf);
printf("Rank %d on core %s of node <%s>\n", rank, clbuf, hnbuf);
}

Answer 2

the exact command seems to be --bind-to l3cache
mpirun --bind-to l3cache -np 16 gmx_mpi mdrun -nt 6