Slower times in multithreading using Julia 1.3.1 - multithreading

Question

I started recently using the new multithreading interface in the 1.3.1 version. After I tried the fibonacci example in this blog post and getting significant speedups, I started experimenting with some old algorithms of mine.
I have a function that uses the trapezoid method to calculate integrals, both below or above a curve:
function trapezoid( x :: AbstractVector ,
y :: AbstractVector ;
y0 :: Number = 0.0 ,
inv :: Number = NaN )
int = zeros(length(x)-1)
for i = 2:length(x)
if isnan(inv) == true
int[i-1] = (y[i]+y[i-1]-2y0) * (x[i]-x[i-1]) / 2
else
int[i-1] = (2inv-(y[i]+y[i-1])-2y0) * (x[i]-x[i-1]) / 2
end # if
end # for
integral = sum(int) ;
return integral
end
Then I have a very inefficient algorithm that determines the midpoint index of a curve comparing the area below and above the curve:
function EAM_without_threads( x :: Vector{Float64} ,
y :: Vector{Float64} ,
y0 :: Real ,
ymean :: Real )
approx = Vector{Float64}(undef,length(x)-1)
for i in 1:length(x)-1
x1 = #view(x[1:i ])
x2 = #view(x[i:end])
y1 = #view(y[1:i ])
y2 = #view(y[i:end])
Al = trapezoid( x1 , y1 , y0=y0 )
Au = trapezoid( x2 , y2 , inv=ymean )
approx[i] = abs(Al-Au)
end
minind = findmin(approx)[2]
return x[minind]
end
And:
function EAM_with_threads( x :: Vector{Float64} ,
y :: Vector{Float64} ,
y0 :: Real ,
ymean :: Real )
approx = Vector{Float64}(undef,length(x)-1)
for i in 1:length(x)-1
x1 = #view(x[1:i ])
x2 = #view(x[i:end])
y1 = #view(y[1:i ])
y2 = #view(y[i:end])
Al = #spawn trapezoid( x1 , y1 , y0=y0 )
Au = #spawn trapezoid( x2 , y2 , inv=ymean )
approx[i] = abs(fetch(Al)-fetch(Au))
end
minind = findmin(approx)[2]
return x[minind]
end
This is what I used to try both functions:
using SpecialFunctions
using BenchmarkTools
x = collect(-10.0:5e-4:10.0)
y = erf.(x)
And then got these results:
julia> #btime EAM_without_threads(x,y,-1.0,1.0)
7.515 s (315905 allocations: 11.94 GiB)
julia> #btime EAM_with_threads(x,y,-1.0,1.0)
10.295 s (1274131 allocations: 12.00 GiB)
I don't understand... Using htop I can see that all my 8 threads are working almost at full capacity. This is my machine:
julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-4712MQ CPU # 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, haswell)
Environment:
JULIA_NUM_THREADS = 8
I know about the overhead of dealing with several threads, and in small problems I understand if it's slower, but why in this case?
I'm also searching for multithreading "good practices", because I guess not every piece of code will benefit from parallelism.
Thank you all in advance.

Answer 1

Your code is doing some very redundant work here. It's doing a full trapezoidal integral for each step, instead of just updating Al and Au incrementally. Here I've rewritten the code so that it does zero allocations, and my version of the EAM is on my computer 5 orders of magnitude faster than the original, without using any threads.
In general: before you start looking into things like threading, consider whether your algorithm is efficient. You can get much bigger speedups from a fast algorithm than from threading.
function trapz(x, y; y0=0.0, inv=NaN)
length(x) != length(y) && error("Input arrays cannot have different lengths")
s = zero(eltype(x))
if isnan(inv)
#inbounds for i in eachindex(x, y)[1:end-1]
s += (y[i+1] + y[i] - 2y0) * (x[i+1] - x[i])
end
else
#inbounds for i in eachindex(x, y)[1:end-1]
s += (2inv - (y[i+1] + y[i]) - 2y0) * (x[i+1] - x[i])
end
end
return s / 2
end
function eam(x, y, y0, ymean)
length(x) != length(y) && error("Input arrays cannot have different lengths")
Au = trapz(x, y; inv=ymean)
Al = zero(Au)
amin = abs(Al - Au)
ind = firstindex(x)
#inbounds for i in eachindex(x, y)[2:end-1] # 2:length(x)-1
Al += (y[i] + y[i-1] - 2y0) * (x[i] - x[i-1]) / 2
Au -= (2ymean - (y[i] + y[i-1])) * (x[i] - x[i-1]) / 2
aval = abs(Al - Au)
if aval < amin
(amin, ind) = (aval, i)
end
end
return x[ind]
end
Benchmarks here (I use #time for your code and #btime for my own, since it would just be too time consuming to use #btime on really slow code):
julia> x = collect(-10.0:5e-4:10.0);
julia> y = erf.(x);
julia> #time EAM_without_threads(x, y, -1.0, 1.0)
15.611004 seconds (421.72 k allocations: 11.942 GiB, 11.73% gc time)
0.0
julia> #btime eam($x, $y, -1.0, 1.0)
181.991 μs (0 allocations: 0 bytes)
0.0
A small extra remark: you should not write if isnan(inv) == true, that is redundant. Just write if isnan(inv).

Answer 2

Try this
function EAM_with_threads( x :: Vector{Float64} ,
y :: Vector{Float64} ,
y0 :: Real ,
ymean :: Real )
approx = Vector{Float64}(undef,length(x)-1)
Threads.#threads for i in 1:length(x)-1
x1 = #view(x[1:i ])
x2 = #view(x[i:end])
y1 = #view(y[1:i ])
y2 = #view(y[i:end])
Al = trapezoid( x1 , y1 , y0=y0 )
Au = trapezoid( x2 , y2 , inv=ymean )
approx[i] = abs(Al-Au)
end
minind = findmin(approx)[2]
return x[minind]
end
Your for loop is easily parallelize, so the lowest fruit is to do each iteration of the "for loop" in parallel. It is much easier to reduce the overall time taken by doing this in parallel then to try and parallelize the internal instance of a "for loop".
I know about the overhead of dealing with several threads, and in
small problems I understand if it's slower, but why in this case?
Well, I think your first problem is that you didn't benchmark how long it takes to do
trapezoid( x1 , y1 , y0=y0 )
If you did, you will find that it takes hardly any time at all. Anything that does not take up a substantial amount of time is not worth doing in parallel. If A and B is independent and they both take up a long time then you should do A and B in parallel. Otherwise find something else to parallelize first.
Lets look at what you have
x = collect(-10.0:5e-4:10.0)
and
for i in 1:length(x)-1
So basically your for loop has around 40000 iterations
Your multithreading method takes
total_time = setup_time * 40000 + ind_work_time/2 * 40000
Where as parallelizing the for loop takes
total_time = setup_time * 1 + ind_work_time * 40000/8
For comparison, the non-multithreaded method take
total_time = ind_work_time * 40000