Multithreaded performance (with Rust) - multithreading

Question

I have been running the following experiment to test multi-threaded performance in Rust.
The below piece of code basically does the following :
STEP 1: Generate 50 million random (key, value) pairs on the main thread.
STEP 2: Process the 50 million pairs by inserting them in a HashMap. This processing step is repeated simultaneously over count threads. Every thread has it's own HashMap.
use rand::Rng;
use std::sync::Arc;
use std::thread;
use std::collections::HashMap;
use std::time::Instant;
fn generate_values(count: usize) -> Vec<([u8; 3], u8)>{
let mut generator = rand::thread_rng();
let mut values = Vec::new();
for _ in 0..count {
let key = generator.gen::<[u8; 3]>();
let value = generator.gen::<u8>();
values.push((key, value));
}
values
}
fn process_values(values: &Arc<Vec<([u8; 3], u8)>>, count: usize) {
let mut handles = Vec::new();
for _ in 0..count {
let values = Arc::clone(values);
handles.push(thread::spawn(move || {
let mut map = HashMap::new();
for (key, value) in values.iter() {
map.insert(key, value);
}
}));
}
for handle in handles {
handle.join().unwrap();
}
}
fn main() {
let values = Arc::new(generate_values(50000000));
println!("processing values...");
for count in 1..=16 {
let start = Instant::now();
process_values(&values, count);
println!("processing values, repeated over {} thread(s), took {:?}", count, start.elapsed());
}
}
I am running the code on a dedicated server with one AMD Ryzen 7 3700X 8-Core processor and 64 GB of RAM, running Ubuntu 20.04. Nothing else is running on the server.
I would have expected that running the code repeated over 1 to 8 threads would have taken roughly the same amount of time, but it seems that running the code on 8 threads took about 23% more time (19.59s) than running it over 1 thread (15.97s) :
processing values, repeated over 1 thread(s), took 15.970677367s
processing values, repeated over 2 thread(s), took 15.936398062s
processing values, repeated over 3 thread(s), took 16.497970587s
processing values, repeated over 4 thread(s), took 17.233953355s
processing values, repeated over 5 thread(s), took 17.233057743s
processing values, repeated over 6 thread(s), took 18.223844841s
processing values, repeated over 7 thread(s), took 19.094954912s
processing values, repeated over 8 thread(s), took 19.592578442s
processing values, repeated over 9 thread(s), took 21.152438731s
processing values, repeated over 10 thread(s), took 22.881476672s
processing values, repeated over 11 thread(s), took 22.97713133s
processing values, repeated over 12 thread(s), took 23.841287249s
processing values, repeated over 13 thread(s), took 24.713425745s
processing values, repeated over 14 thread(s), took 24.979827585s
processing values, repeated over 15 thread(s), took 25.78961309s
processing values, repeated over 16 thread(s), took 26.511473666s
Then I thought that it had something to do with hyper-threading, so I disabled simultaneous multi-threading :
echo off > /sys/devices/system/cpu/smt/control
These are the results without hyper-threading :
processing values, repeated over 1 thread(s), took 15.906120824s
processing values, repeated over 2 thread(s), took 15.927443081s
processing values, repeated over 3 thread(s), took 16.701871709s
processing values, repeated over 4 thread(s), took 16.73429606s
processing values, repeated over 5 thread(s), took 17.785883476s
processing values, repeated over 6 thread(s), took 18.171144237s
processing values, repeated over 7 thread(s), took 18.871619003s
processing values, repeated over 8 thread(s), took 19.439770035s
processing values, repeated over 9 thread(s), took 22.937699259s
processing values, repeated over 10 thread(s), took 25.164055752s
processing values, repeated over 11 thread(s), took 29.44375459s
processing values, repeated over 12 thread(s), took 30.436276538s
processing values, repeated over 13 thread(s), took 33.775704733s
processing values, repeated over 14 thread(s), took 35.962573012s
processing values, repeated over 15 thread(s), took 38.04670196s
processing values, repeated over 16 thread(s), took 40.535251291s
Still the same unexpected 22-23% performance decrease when going from 1 to 8 threads.
Although the performance when going from 8 to 16 threads is consistent and as expected (it takes about twice the time to run the code on 16 threads versus 8 threads).
Here is a little graph of the relative time taken to run the code on count threads versus 1 thread.
Is such a performance decrease of 22-23% expected when repeating the code over 8 threads versus 1 thread, when run on an 8-core processor ?
In other words, what explains the performance decrease ?
Code is run in release with "cargo run --release".

Answer 1

Two possibilities:
Even without lock contention memory and caches are a shared resource and speed when running more code in parallel is constrained on this resource. You can get some clues with perf stat, e.g. if there are more cache misses.
Modern processors run with a higher frequency if only a few cores are active. For the Ryzen / 3700 X in turbo mode the CPU runs with up to 4.4GHz while the base clock rate is only 3.6GHz.