Reassembling chunks of a binary file in Perl in a multithreaded script - multithreading

Question

I'm trying to break a binary file into chunks for a multipart upload and then have them reassembled correctly. I'm using threads to try to speed up transfers. The result hasn't validated using md5sum so I decided to try to just to reassemble it locally and see what happens. The local result comes out ok with a small text file, but not a large binary file, when multithreaded. It works either way when single threaded.
Here's a simple (admittedly inefficient) implementation of a single threaded version that works:
use strict;
use POSIX;
my $file = 'test.txt';
my $chunkSize = 3;
my $size = -s $file;
my $parts = ceil($size / $chunkSize);
my %output;
open(my $fileHandle, '<', $file) or die("Error reading file, stopped");
binmode($fileHandle);
for (my $i = 0; $i < $parts; $i++) {
my $chunk;
my $offset = $i * $chunkSize;
seek ($fileHandle, $offset, 0);
if ($chunkSize > $size) {
$chunkSize = $size;
}
elsif (($chunkSize + $offset) > $size) {
$chunkSize = $size - $offset;
}
my $length = read($fileHandle, $chunk, $chunkSize);
say STDERR "Chunk size for $i: " . $chunkSize . "(" . ($chunkSize + $offset) . "/$size) starting at " . $offset .": ";# . $chunk;
$output{ $i } = $chunk;
}
close($fileHandle);
open(my $fileHandle, '>', $file . '.comp') or die("Error reading file, stopped");
binmode($fileHandle);
for (1...$parts) {
print $fileHandle $output{ $_ };
}
close ($fileHandle);
Here's the multithreaded version that does not:
use strict;
use POSIX;
use threads;
use threads::shared;
use Thread::Queue;
my $file = '/backup/2022-12-13/accounts/test.tar.gz';
my $chunkSize = 500 * 1024 * 1024;
#Or my $chunkSize = 3; when using it with the smaller text test file.
my $size = -s $file;
my $parts = ceil($size / $chunkSize);
my %output :shared;
my #queueData;
for (0...$parts) { push (#queueData, $_); }
my $overallProgress :shared;
my $queue = Thread::Queue->new(#queueData);
open(my $fileHandle, '<', $file) or die("Error reading file, stopped");
binmode($fileHandle);
my #threads;
for(1..4) {
push #threads, threads->create( sub {
while( my $i = $queue->dequeue_nb ) {
my $chunk;
my $offset = ($i - 1) * $chunkSize;
seek ($fileHandle, $offset, 0);
if ($chunkSize > $size) {
$chunkSize = $size;
}
elsif (($chunkSize + $offset) > $size) {
$chunkSize = $size - $offset;
}
my $length = read($fileHandle, $chunk, $chunkSize);
say STDERR "Chunk size for $i: " . $chunkSize . "(" . ($chunkSize + $offset) . "/$size) starting at " . $offset .": ";
$output{ $i } = $chunk;
}
});
}
$_->join for #threads;
close($fileHandle);
open(my $fileHandle, '>', $file . '.comp') or die("Error reading file, stopped");
binmode($fileHandle);
for (1...$parts) {
print "Reassembling part $_...";
print $fileHandle $output{ $_ };
}
close ($fileHandle);
If I run that code on this simple text file, it works:
1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
Result of md5sum on both source and destination file:
6d8d0b2c6a06f88e13167fffacb1b74e /home/tbutler/text.txt
6d8d0b2c6a06f88e13167fffacb1b74e /home/tbutler/text.txt.comp
But, if I take a 9 GB tarball (and increase the chunk size to 500 MB by changing the appropriate line to my $chunkSize = 500 * 1024 * 1024) and run it likewise, the md5sum result is not matching. In fact the resultant file is approximately a gigabyte smaller than the original file and I can't figure out why.