Perl: build complex object tree using multiple threads - multithreading

Question

I am trying to optimize code that loads a file and parses the data in Perl.
Background
The data ends up in a complicated object tree.
The top level object is a blessed package.
Some of the nested objects are new'ed as another blessed package type call Item.
The first pass separates the binary data into units and segments
within a unit which are all stored in multiple arrays.
There could be 20 or 50 units with 8 segments each.
The second pass performs the decoding of the binary data and is what needs optimized for speed.
Approach with threads
I am trying to use the modules threads and threads::shared.
I want each thread to process a subset of the units and populate the data into a common object tree.
I am looking for some sample code that demonstrates how to share blessed objects that may be allocated and blessed from any thread context and inserted into a shared object tree. And then is accessible from the main thread to walk the data for data lookups. The decoding threads will return once the decoding is completed.
I am having difficulty seeing how to have multiple threads insert objects to a common object tree using the threads:shared module. In particular when object of Item type are blessed from a thread context. The class (package) functions won't be bound to the object as I understand.
I do realize that at certain points in the code the code will need to use the threads::shared::lock() function before adding objects to the object tree.
In particular the nested blessed Item objects would be allocated from each thread context.
The threads::shared documentation says "Note that it is often not wise to share an object unless the class itself has been written to support sharing".
Is there a sample code that demonstrates how to accomplish this?
The documentation also says "object's destructor may get called multiple times, one for each thread's scope exit". How is this handled properly?
Thanks
J.R.

Answer 1

OK, so backtracking a bit - threads::shared really does 'single data structures' and doesn't really support more complicated things. That's because when you 'thread' you actually create separate program instances with (some) shared memory space, but practically speaking each 'thread' is a separate program anyway.
So, supporting sharing in an object gets really quite messy. I've found a better approach is to ... not. Use Thread::Queue to pass data between threads, and have one thread that acts to collate results. If you need to pass more complicated data structures, you can use Storable and freeze/thaw to serialise the object, and enqueue it.
That way you don't have to worry about tripping over shared nested data structures - and it's quite likely that you will, because there's no such thing as a 'deep share' option on an object - you have to explicitly share every internal array/hash(reference).
So I would tackle it like this:
#!/usr/bin/perl
use strict;
use warnings;
package Test_Object;
sub new {
my ( $class, $id ) = #_;
my $self = {};
$self->{id} = $id;
bless $self, $class;
return $self;
}
sub set_result {
my ( $self, $result_code ) = #_;
$self->{result} = $result_code;
}
sub get_id {
my ($self) = #_;
return $self->{id};
}
sub get_result {
my ($self) = #_;
return $self->{result};
}
package main;
use strict;
use warnings qw/ all /;
use threads;
use Thread::Queue;
use Storable qw/ freeze thaw/;
my $work_q = Thread::Queue->new();
my $result_q = Thread::Queue->new();
sub worker {
my $tid = threads->self->tid;
print "$tid: starting\n";
while ( my $item = $work_q->dequeue() ) {
my $object = thaw($item);
print "$tid: got object with ID of ", $object->get_id, "\n";
$object->set_result( $object->get_id . " : $tid" );
$result_q->enqueue( freeze $object );
}
}
sub collator {
while ( my $result = $result_q->dequeue ) {
my $object = thaw $result;
print "Collator got object with result code of ", $object->get_result,
"\n";
}
## do something with collated wossnames - pass back to main maybe?
}
my #workers;
for ( 1 .. 5 ) {
my $thr = threads->create( \&worker );
push #workers, $thr;
}
my $collator = threads->create( \&collator );
for ( 1 .. 200 ) {
my $work_object = Test_Object->new($_);
$work_q->enqueue( freeze $work_object );
}
$work_q->end;
foreach my $thr (#workers) {
$thr->join;
}
$result_q->end;
foreach my $thr ( threads->list ) {
$thr->join;
}