Threaded script exits prematurely - multithreading

Question

I want to do something along the lines of:
my $sema = Thread::Semaphore->new(8);
while(#compsCopy)
{
my $thread1 = threads->create('Build', (shift #compsCopy), "clean");
}
sub FfsBuild {
$sema->down();
my ($comp, $action) = #_;
my $cmd = "$MAKE $MAKE_INVOCATION_PATH/$comp $action";
my $retCode = system($cmd);
push(#retCodes, $retCode);
print "\n\t\t**** ERROR IN $comp ****\n" unless $retCode == 0;
print "added proc $comp\n";
$sema->up();
return $retCode;
}
This seems to work fine for the first 10-20 directories... but eventually I get:
Perl exited with active threads:
364 running and unjoined
14 finished and unjoined
0 running and detached
C:\dev>make: Leaving directory `/cygdrive/C/dev/dir0'
make: Leaving directory `/cygdrive/C/dev/dir1'
make: Leaving directory `/cygdrive/C/dev/dir2'
make: Leaving directory `/cygdrive/C/dev/dir3'
make: Leaving directory `/cygdrive/C/dev/dir4'
make: Leaving directory `/cygdrive/C/dev/dir5'
make: Leaving directory `/cygdrive/C/dev/dir6'
make: Leaving directory `/cygdrive/C/dev/dir7'
Where am I messing up here?

Answer 1

The problem is that you don't wait for you your threads to finish. Adding the following at the bottom will do the trick:
$_->join for threads->list;
You shouldn't create 378 threads then make it so only 8 execute at a time! What a waste! It's people running code like that that say Perl threads are inefficient. Create a pool of workers and assign the work to them.
use constant NUM_WORKERS => 8;
use Thread::Queue 3.01 qw( );
sub worker {
my ($job) = #_;
FfsBuild($job, 'clean');
}
{
my $q = Thread::Queue->new();
for (1..NUM_WORKERS) {
async {
while (defined(my $job = $q->dequeue()) {
worker($job);
}
};
}
$q->enqueue($_) for #compsCopy;
# When you're done adding to the queue.
$q->end();
$_->join() for threads->list();
}
FfsBuild is what you had without the semaphore.

Answer 2

Threading with Perl is a bit difficult (and also rather inefficient. Don't do it without a good reason).
Every thread except the main thread either has to be joined prior to exit, or be detached. You should still make sure that the thread terminates before you exit the process.
So something like threads->create(...)->detach should work here, but I think it's better to join:
my #threads;
for my $job (#jobs) {
push #threads, threads->create(\&worker, $job);
}
# wait until all have completed
$_->join for #threads;
However, you will probably want to kick of a few workers at the start of the program, and feed them jobs via a Thread::Queue instead – each thread is a complete clone of the current interpreter state, which tends to use unnecessary large amounts of memory. Running hundreds of Perl threads is not likely to be useful.