Parallel read file with dask - multithreading

Question

I have a question about delayed decoration, it may seem similar to the following question" Dask: How would I parallelize my code with dask delayed? "
but even there it is not answered. I have the following code :
#dask.delayed
def remove_unnessasey_data(temp,l1):
do some work
return temp
#dask.delayed
def change_structure(temp):
do some work
return temp1
#dask.delayed
def read_one(filename):
return pd.read_csv(filename)
and then:
def f(filenames):
results = []
for filename in filenames:
results.append(change_structure( remove_unnessasey_data(
read_one(filename),l1)))
return results
result = dask.compute(*result)
according to this it should increase the speed, but the speed is the same if I read in chunks from the big file can anybody explain why??
I am aware of GIL but according to the documentation it should enhance the speed

Answer 1

according to this it should increase the speed
Bollocks. That documentation, for lack of a better word, is wrong in general.
Saying that doing IO in parallel will increase performance in general displays a significant misunderstanding of how most filesystems and disk storage systems work.
Why?
Seek time.
Generally, filesystems store files in as contiguous chunks as possible. To read position X in the file, the disk heads first have to be positioned over the track that holds the sector X is in. That takes time. Then the system has to wait until that sector rotates under the disk heads. That again takes time.
It should be obvious why reading a file sequentially from a spinning disk is faster - to read sector N, the disk heads have to first seek to the track that contains sector N. But because files are stored as contiguously as possible, the track that contains sector N also likely contains sector N+1, N+2, N+3, and quite a bit more. Toss in the read ahead caching that both the disk (disks are not usually dumb devices - they're pretty much full-fledged IO computers that have built-in cache systems) and the filesystem do, and sequential reading of a file from a spinning disk tends to minimize the time spent looking for data.
Now try reading in parallel.
Thread A reads sector X. Disk seeks to track, waits for sector X to pass under the heads. While that's happening, thread B tries to read sector Y. Disk finally gets to read sector X, but has a pending command to read sector Y. Now disk has to seek heads to the proper track, perhaps abandoning the readahead it would have done to get sector X+1 for thread A's next read, wait for the heads to move, then wait for sector Y to pass under the heads to read.
Meanwhile, thread C issues a request to read sector Z...
And the disk heads dance all over the disk. Then wait for the proper sector to pass under the heads.
A typical consumer-grade 5,400 RPM SATA disk that nominally supports IO rates of 100 MB/sec can be reduced to a few KILOBYTES per second through such IO patterns.
Reading or writing data in parallel almost never increases speed, especially if you're using standard filesystems on spinning disks.
You can get better performance using SSD(s) if a single thread's IO doesn't saturate the storage system - not just the disk, but entire path from CPU to/from disk. Many, many motherboards have cheap, slow disk controllers and/or lack IO bandwidth. How many people completely ignore the disk controller or the IO bandwidth of the motherboard when buying a computer?
There are filesystems that do support parallel IO for improved performance. They tend to be proprietary, expensive, and FAST. IBM's Spectrum Scale (originally GPFS) and Oracle's HSM (originally SAMFS/QFS) are two examples.