Is there a multiprocessing module for Perl? - multithreading

Question

Is there a multiprocessing module for Perl? Something that has similar functionality to what's offered by Python's multiprocessing module.
I understand I could build similar functionality using Perl, but I'm looking for something already implemented.

Answer 1

forks provides the same awesome interface as threads, but uses processes instead of threads.
use forks; # Or: use threads;
use Thread::Queue;
my $q = Thread::Queue->new();
my #workers;
for (1..NUM_WORKERS) {
push #workers, async {
while (defined(my $job = $q->dequeue())) {
...
}
};
}
$q->enqueue(...);
$q->enqueue(undef) for #workers;
$_->join for #workers;
Comparing forks with Forks::Super.
Keep in mind, these are suppose to the be the cases where Forks::Super excels!
use Forks::Super;
sub do_something { my #args = #_; ... }
$process = fork { sub => \&do_something, args => [#args] };
$process->wait;
can be written as
use forks;
sub do_something { my #args = #_; ... }
$process = async { do_something(#args) };
$process->join;
---
use Forks::Super;
my $x = 42;
my #y = ();
my %z = ();
sub do_something_else {
$x = 19;
#y = qw(foo bar);
%z = (foo => 'bar');
}
$process = fork { sub => 'do_something_else', share => [\$x, \#y, \%z ] };
$process->wait;
can be written as
use forks;
use forks::shared;
my $x :shared = 42;
my #y :shared = ();
my %z :shared = ();
sub do_something_else {
$x = 19;
#y = qw(foo bar);
%z = (foo => 'bar');
}
$process = async { do_something_else() };
$process->join;
---
use Forks::Super;
use IO::Handle;
pipe my $child_read, my $parent_write;
pipe my $parent_read, my $child_write;
$parent_write->autoflush(1);
$child_write->autoflush(1);
sub square {
while (my $x = <$child_read>) {
chomp($x);
print {$child_write} $x ** 2, "\n";
}
close $child_write;
}
$process = fork { sub => 'square' };
print { $parent_write } "9\n";
chomp( my $result = <$parent_read> ); # 81
close $parent_write;
$process->wait;
can be written as
use forks;
use Threads::Queue;
my $req = Threads::Queue->new();
my $resp = Threads::Queue->new();
sub square { $_[0] ** 2 }
$process = async {
while (defined(my $x = $req->dequeue())) {
$resp->enqueue( square($x) );
}
};
$req->enqueue(9);
my $result = $resp->dequeue(); # 81
$resp->enqueue(undef);
$process->join;
---
use Forks::Super;
sub square_root {
sleep 1 && seek STDIN,0,1 while eof(STDIN); # ok, this is a workaround for an existing bug :-(
while (my $x = <STDIN>) {
chomp($x);
print sqrt($x), "\n";
}
}
$process = fork { sub => 'square_root', child_fh => 'in,out,block' };
$process->write_stdin("81\n");
chomp( $result = $process->read_stdout() ); # 9
$process->close_fh('stdin');
$process->wait;
can be written as
use forks;
use Threads::Queue;
my $req = Threads::Queue->new();
my $resp = Threads::Queue->new();
$process = async {
while (defined(my $x = $req->dequeue())) {
$resp->enqueue( sqrt($x) );
}
};
$req->enqueue(81);
my $result = $resp->dequeue(); # 9
$resp->enqueue(undef);
$process->join;

Answer 2

I think Forks::Super comes pretty close. It has a few features for running an arbitrary subroutine (or external command) in a background process, monitoring and signalling the background process, and making interprocess communication a little less painful.
use Forks::Super;
sub do_something { my #args = #_; ... }
$process = fork { sub => \&do_something, args => [#args] };
$process->wait;
my $x = 42;
my #y = ();
my %z = ();
sub do_something_else {
$x = 19;
#y = qw(foo bar);
%z = (foo => 'bar');
}
$process = fork { sub => 'do_something_else', share => [\$x, \#y, \%z ] };
$process->wait;
# $x, #y, and %z are now updated with changes made in background process
# create your own pipes to use for IPC
use IO::Handle;
pipe my $child_read, my $parent_write;
pipe my $parent_read, my $child_write;
$parent_write->autoflush(1);
$child_write->autoflush(1);
sub square {
while (my $x = <$child_read>) {
print {$child_write} $x ** 2, "\n";
}
close $child_write;
}
$process = fork { sub => 'square' };
print {$parent_write} "9\n";
my $result = <$parent_read>; # should be "81\n";
close $parent_write;
# or use the standard I/O handles for IPC
sub square_root {
sleep 1 && seek STDIN,0,1 while eof(STDIN); # ok, this is a workaround for an existing bug :-(
while (my $x = <STDIN>) {
print sqrt($x), "\n";
}
}
$process = fork { sub => 'square_root', child_fh => 'in,out,block' };
$process->write_stdin("81\n");
$result = $process->read_stdout(); # => "9\n"
Both the multiprocessing module and Forks::Super have a lot of features. Which ones are you specifically interested in?
I am the author of Forks::Super and my goal is to include any features for parallel processing that people find useful, so if there's a feature in multiprocessing that you want in Perl, let me know.

Answer 3

What about POE: Perl Object Environment? It has support for asynchronous child processes.

Answer 4

You can use https://github.com/marioroy/mce-perl
It is similar to python multiprocess module