How to share socket hash to thread function in perl? - multithreading

I have tried to share hash value to thread function. The hash values contain soket connection , But I didn't share the hash variable to thread function. I am getting Invalid value for shared scalar error. But when I assign normal text or integer values to the hash , that time the hash variable able to share to thread function. Can any one please solve the problem ? Below code I have tried
use strict;
use Data::Dumper;
use IO::Socket;
use IO::Select;
use JSON ;
use threads;
use threads::shared;
my $i=5 ;
my ($Main_Socket_Hash,$Read_Client );
$|=1;
print "Calling" ;
my ($Chat_Details ) ;
($main::Server,$main::Port)=("192.168.12.37","6666");
$main::Socket= IO::Socket::INET->new(
'LocalAddr' => $main::Server,
'LocalPort' => $main::Port,
'Proto' => 'tcp',
'Listen' => '120' ,
'Reuse' => 1
) ;
Handle_Client() ;
sub Handle_Client {
my ( $New_Socket,$New_Client,#Ready_Clints,$Thread_Id ) ;
share($i) ;
shared($Main_Socket_Hash) ;
$Thread_Id=threads->create(\&Read_Outgoing_Message);
$Read_Client = new IO::Select() ;
$Read_Client->add($main::Socket) ;
while ( 1 ) {
print "Main loop \n" ;
my ($Read_Sockets) = IO::Select->select($Read_Client, undef, undef, 0);
foreach $New_Socket (#$Read_Sockets) {
Process_Request($New_Socket) ;
}
sleep(1);
$i=$i+5 ;
}
}
sub Process_Request {
my ( $New_Socket,$New_Client,$Request_Api ) =#_ ;
if ( $New_Socket eq $main::Socket ) {
$New_Client=$New_Socket->accept();
$Read_Client->add($New_Client ) ;
}
else {
$Request_Api=<$New_Socket> ;
unless ( $Request_Api ) {
$Read_Client->remove($New_Socket) ;
close($New_Socket)
}
else {
chomp($Request_Api );
Validation_Request_Api($New_Socket,$Request_Api) ;
}
}
}
sub Validation_Request_Api {
my ( $New_Socket, $Request_Api ,$Request_Data ) = #_ ;
$Request_Data=decode_json($Request_Api) ;
unless ( $Request_Data )
{
return "FAILURE" ;
}
unless ( $Main_Socket_Hash->{$Request_Data->{'chatId'}} )
{
$Main_Socket_Hash->{$Request_Data->{'chatId'}}=$New_Socket;
#$Main_Socket_Hash->{$Request_Data->{'chatId'}}=$Request_Data->{'chatId'};
}
return "SUCCESS" ;
}
sub Read_Outgoing_Message {
my ( $Status,$Outgoing_Message ) ;
while ( 1 ) {
print "%%%%%%%%%%%%%%%%%%%===> $i <=====%%%%%%%%%%%%%\n" ;
print "%%%%%%%%%%%%%%%%%%%===> " . Dumper ( $Main_Socket_Hash) ." <=====%%%%%%%%%%%%%\n" ;
sleep ( 1) ;
}
}
Note :
When I get error while assigning socket connection to hash value
$Main_Socket_Hash->{$Request_Data->{'chatId'}}=$New_Socket;

Related

Perl - multithreading on foreach loop:

I have a big code that somewhere in the middle of it, there's a foreach loop filling up some dictionaries. Each iteration is independant and fills those dictionaries with disjoint keys.
I'm trying to turn the "foreach" loop in the middle to multithreaded in order to decrease time.
In the following example, $a1, $b1 are the pointers to dictionaries.
I tried "thread::shared" this way:
my $a1 = {};
my $b1 = {};
my $c1 = {};
my $d1 = {};
# a lot of code using $a1 and $b1
share($a1);
share($b1);
share($c1);
share($d1);
my #threads;
foreach my $Inst ( sort keys %{ $a1->{ports} }) {
push( #threads, threads->create('some_func', $Inst, $a1, $b1, $c1, $d1, $e ...));
}
for my $thr (#threads) {
thr->join();
}
# all the other code
But I get an error of:
Invalid value for shared scalar at ...
Any ideas how to get the data-structures filled, but not that it would interfere with the code before and after the for-each loop?
It is not possible to make a hash shared after it has been created/declared without losing the data in the hash. Instead you could try use shared_clone() like this:
use feature qw(say);
use strict;
use warnings;
use threads ;
use threads::shared ;
use Data::Dumper qw(Dumper);
my %h1 = (a => 1, b => 2);
my %h2 = (c => 3, d => 4);
my $a1 = \%h1;
my $b1 = \%h2;
my $a1c = shared_clone($a1);
my $b1c = shared_clone($b1);
my $lockvar:shared;
my $nthreads = 3;
for ( 1..$nthreads ) {
threads->create('job_to_parallelize', $a1c, $b1c, \$lockvar ) ;
}
$_->join() for threads->list();
sub job_to_parallelize {
my ($a1, $b1, $lockvar) = #_;
{
lock $lockvar;
$a1->{a}++;
$b1->{d}++;
}
}
print Dumper({a1c => $a1c});
print Dumper({b1c => $b1c});
Output:
$VAR1 = {
'a1c' => {
'a' => 4,
'b' => 2
}
};
$VAR1 = {
'b1c' => {
'd' => 7,
'c' => 3
}
};

Passing a thread queue object as a object variable in perl

I am writing a package which takes inputs like thread count,thread::Queue objects.
Once the package object is created, I will create the threads based on the input argument and dequeue the input queue in each thread and each threads executes a simple unix command like pinging to a server(changed to keep it simple).
Below is the code:
my $failed_q = Thread::Queue -> new();
my $success_q = Thread::Queue -> new();
my $process_q = Thread::Queue -> new();
package WorkerThreads;
sub new {
my $class = shift;
my $self = {
_ThreadCount => shift,
_FidQueue => shift,
_SuccessQueue => shift,
_FailedQueue => shift,
};
bless $self, $class;
return $self;
}
sub WorkerProcess
{
my ($self)=#_;
while ( my $fid = $self->{_FidQueue} -> dequeue() )
{
chomp ( $fid );
print threads -> self() -> tid(). ": pinging $fid\n";
my $result = `/bin/ping -c 1 $fid`;
if ( $? ) { $self->{_FailedQueue} -> enqueue ( $fid ) }
else { $self->{_SuccessQueue} -> enqueue ( $fid ) ; }
sleep 1;
}
print threads -> self() -> tid(). ":\n";
}
sub CreateThreads
{
my ($self)=#_;
my $Num_of_threads=$self->{_ThreadCount};
for ( 1..$Num_of_threads )
{
threads -> create ( \&WorkerProcess );
}
}
sub StartThreads
{
my ($self)=#_;
foreach my $thr ( threads -> list() )
{
$thr -> join();
}
}
sub PrintResult
{
my ($self)=#_;
while ( my $fid = $self->{_FailedQueue} -> dequeue_nb() )
{
print "$fid failed to ping\n";
}
#collate results. ('synchronise' operation)
while ( my $fid = $self->{_SuccessQueue} -> dequeue_nb() )
{
print "$fid Ping Succeeded\n";
}
}
sub ProcessRequest
{
my ($self)=#_;
$self->CreateThreads(#_);
$self->StartThreads(#_);
$self->PrintResult(#_);
}
package main;
#insert tasks into thread queue.
open ( my $input_fh, "<", "server_list" ) or die $!;
$process_q->enqueue( <$input_fh> );
close ( $input_fh );
my $Workers;
$Workers=WorkerThreads->new(
10,
$process_q,
$success_q,
$failed_q
);
$Workers->ProcessRequest();
I am getting an error while i try to dequeue in the while loop that queue is undefined. So i got this doubt like can we pass a thread queue object to a package as an argument.
Thread 1 terminated abnormally: Can't call method "dequeue" on an undefined value at .
You're passing a function into threads->create without any context of which object they are working within. Something like this should work
threads->create( sub { $self->WorkerProcess } );

How to pause and resume a multithread perl script?

I have written the perl script to pause and resume.When the user enters Ctrl+c it has to pause and on pressing c it should resume. But is not working properly as expected. Can anyone help me on this what mistake i am making:
use strict;
use threads;
use threads::shared;
use Thread::Suspend;
use Lens;
$SIG{'INT'} = 'Pause';
#$| = 1;
print chr(7);
my $nthreads = 64;
my #thrs;
for(1..$nthreads)
{
print "START $_ \n";
my ($thr) = threads->create(\&worker, $_);
push #thrs ,$thr;
}
$_->join for #thrs;
exit;
sub worker
{
my $id = shift;
my $tmp;
my $lens = Lens->new("172.16.1.65:2000");
die "cannot create object" unless defined $lens;
die "cannot connect to XRay at " unless defined $lens->open("172.16.1.65:2000");
for(1..100000)
{
print "Thread $id \n";
}
print "$id>LOAD EXIT\n";
}
sub Pause
{
sleep(1);
print "\nCaught ^C\n";
print "Press \"c\" to continue, \"e\" to exit: ";
$_->suspend() for #thrs;
while (1)
{
my $input = lc(getc());
chomp ($input);
if ($input eq 'c') {
#clock($hour,$min,$sec);
$_->resume() for #thrs;
return;
}
elsif ($input eq 'e') {
exit 1;
}
}
}
Well, you haven't been too specific as to how it's "not working properly". But I would suggest looking at using Thread::Semaphore for a 'suspend' mechanism.
I would also suggest not using signal and instead doing something like:
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Thread::Semaphore;
use Term::ReadKey;
my $nthreads = 64;
my $thread_semaphore = Thread::Semaphore->new($nthreads);
sub worker {
for ( 1 .. 10 ) {
$thread_semaphore->down();
print threads->self->tid(), "\n";
sleep 1;
$thread_semaphore->up();
}
}
for ( 1 .. $nthreads ) {
threads->create( \&worker );
}
my $keypress;
ReadMode 4;
while ( threads->list(threads::running) ) {
while ( not defined( $keypress = ReadKey(-1) )
and threads->list(threads::running) )
{
print "Waiting\nRunning:". threads->list(threads::running) . "\n";
sleep 1;
}
print "Got $keypress\n";
if ( $keypress eq "p" ) {
print "Pausing...";
$thread_semaphore -> down_force($nthreads);
print "All paused\n";
}
if ( $keypress eq "c" ) {
print "Resuming...";
$thread_semaphore -> up ( $nthreads );
}
}
ReadMode 0;
foreach my $thr ( threads->list ) {
$thr->join();
}
It'll 'suspend' by setting the semaphores to zero (or negative) and relies on the threads checking if they should be stopping here or not.
I think the root of your problem though, will probably be signal propagation - your signal handler is global across your threads. You might find configuring $SIG{'INT'} for your threads separately will yield better results. (E.g. set the signal handler to 'IGNORE' at the start of your code, and set specific ones in the thread/main once the threads have been spawned).

Perl thread performance with a shared variable

I'm working on a project, implemented in Perl, and thought it would be an idea to use threads to distribute the work, because the tasks can be done independent of each other and only reading from shared data in memory. However, the performance is nowhere near as I expect it to be. So after some investigation I can only conclude that threads in Perl basically suck, but I keep wondering the performance goes down the drain as soon as I implement one single shared variable.
For example, this little program has nothing shared and consumes 75% of the CPU (as expected):
use threads;
sub fib {
my ( $n ) = #_;
if ( $n < 2 ) {
return $n;
} else {
return fib( $n - 1 ) + fib( $n - 2 );
}
}
my $thr1 = threads->create( 'fib', 35 );
my $thr2 = threads->create( 'fib', 35 );
my $thr3 = threads->create( 'fib', 35 );
$thr1->join;
$thr2->join;
$thr3->join;
And as soon as I introduce a shared variable $a, the CPU usage is somewhere between 40% and 50%:
use threads;
use threads::shared;
my $a : shared;
$a = 1000;
sub fib {
my ( $n ) = #_;
if ( $n < 2 ) {
return $n;
} else {
return $a + fib( $n - 1 ) + fib( $n - 2 ); # <-- $a was added here
}
}
my $thr1 = threads->create( 'fib', 35 );
my $thr2 = threads->create( 'fib', 35 );
my $thr3 = threads->create( 'fib', 35 );
$thr1->join;
$thr2->join;
$thr3->join;
So $a is read-only and no locking takes place, and yet the performance decreases. I'm curious why this happens.
At the moment I'm using Perl 5.10.1 under Cygwin on Windows XP. Unfortunately I couldn't test this on a non-Windows machine with a (hopefully) more recent Perl.
Your code is a tight loop around a synchronized structure. Optimize it by having each thread copy the shared variable -- just once for each thread -- into an unshared variable.
Constructing a shared object containing lots of data is possible in Perl and not worry about extra copies. There is no impact to performance when spawning workers, because the shared data resides inside a separate thread or process, depending on whether using threads.
use MCE::Hobo; # use threads okay or parallel module of your choice
use MCE::Shared;
# The module option constructs the object under the shared-manager.
# There's no trace of data inside the main process. The construction
# returns a shared reference containing an id and class name.
my $data = MCE::Shared->share( { module => 'My::Data' } );
my $b;
sub fib {
my ( $n ) = #_;
if ( $n < 2 ) {
return $n;
} else {
return $b + fib( $n - 1 ) + fib( $n - 2 );
}
}
my #thrs;
push #thrs, MCE::Hobo->create( sub { $b = $data->get_keys(1000), fib(35) } );
push #thrs, MCE::Hobo->create( sub { $b = $data->get_keys(2000), fib(35) } );
push #thrs, MCE::Hobo->create( sub { $b = $data->get_keys(3000), fib(35) } );
$_->join() for #thrs;
exit;
# Populate $self with data. When shared, the data resides under the
# shared-manager thread (via threads->create) or process (via fork).
package My::Data;
sub new {
my $class = shift;
my %self;
%self = map { $_ => $_ } 1000 .. 5000;
bless \%self, $class;
}
# Add any getter methods to suit the application. Supporting multiple
# keys helps reduce the number of trips via IPC. Serialization is
# handled automatically if getter method were to return a hash ref.
# MCE::Shared will use Serial::{Encode,Decode} if available - faster.
sub get_keys {
my $self = shift;
if ( wantarray ) {
return map { $_ => $self->{$_} } #_;
} else {
return $self->{$_[0]};
}
}
1;

Threads application terminates unexpectedly

I have little scraping application and trying to add multithreading to it. Here is code (MyMech is WWW::Mechanize subclass used to process HTTP errors):
#!/usr/bin/perl
use strict;
use MyMech;
use File::Basename;
use File::Path;
use HTML::Entities;
use threads;
use threads::shared;
use Thread::Queue;
use List::Util qw( max sum );
my $page = 1;
my %CONFIG = read_config();
my $mech = MyMech->new( autocheck => 1 );
$mech->quiet(0);
$mech->get( $CONFIG{BASE_URL} . "/site-map.php" );
my #championship_links =
$mech->find_all_links( url_regex => qr/\d{4}-\d{4}\/$/ );
foreach my $championship_link (#championship_links) {
my #threads;
my $queue = Thread::Queue->new;
my $queue_processed = Thread::Queue->new;
my $url = sprintf $championship_link->url_abs();
print $url, "\n";
next unless $url =~ m{soccer}i;
$mech->get($url);
my ( $last_round_loaded, $current_round ) =
find_current_round( $mech->content() );
unless ($last_round_loaded) {
print "\tLoading rounds data...\n";
$mech->submit_form(
form_id => "leagueForm",
fields => {
round => $current_round,
},
);
}
my #match_links =
$mech->find_all_links( url_regex => qr/matchdetails\.php\?matchid=\d+$/ );
foreach my $link (#match_links) {
$queue->enqueue($link);
}
print "Starting printing thread...\n";
my $printing_thread = threads->create(
sub { printing_thread( scalar(#match_links), $queue_processed ) } )
->detach;
push #threads, $printing_thread;
print "Starting threads...\n";
foreach my $thread_id ( 1 .. $CONFIG{NUMBER_OF_THREADS} ) {
my $thread = threads->create(
sub { scrape_match( $thread_id, $queue, $queue_processed ) } )
->join;
push #threads, $thread;
}
undef $queue;
undef $queue_processed;
foreach my $thread ( threads->list() ) {
if ( $thread->is_running() ) {
print $thread->tid(), "\n";
}
}
#sleep 5;
}
print "Finished!\n";
sub printing_thread {
my ( $number_of_matches, $queue_processed ) = #_;
my #fields =
qw (
championship
year
receiving_team
visiting_team
score
average_home
average_draw
average_away
max_home
max_draw
max_away
date
url
);
while ($number_of_matches) {
if ( my $match = $queue_processed->dequeue_nb ) {
open my $fh, ">>:encoding(UTF-8)", $CONFIG{RESULT_FILE} or die $!;
print $fh join( "\t", #{$match}{#fields} ), "\n";
close $fh;
$number_of_matches--;
}
}
threads->exit();
}
sub scrape_match {
my ( $thread_id, $queue, $queue_processed ) = #_;
while ( my $match_link = $queue->dequeue_nb ) {
my $url = sprintf $match_link->url_abs();
print "\t$url", "\n";
my $mech = MyMech->new( autocheck => 1 );
$mech->quiet(0);
$mech->get($url);
my $match = parse_match( $mech->content() );
$match->{url} = $url;
$queue_processed->enqueue($match);
}
return 1;
}
And i have some strange things with this code. Sometimes it run but sometimes it exit with no errors (at the ->detach point). I know that #match_links contain data but threads are not created and it just close. Usually it terminates after processing second $championship_link entry.
May be i'm doing something wrong?
Update
Here is code for find_current_round subroutine (but i'm sure it's not related to the question):
sub find_current_round {
my ($html) = #_;
my ($select_html) = $html =~ m{
<select\s+name="round"[^>]+>\s*
(.+?)
</select>
}isx;
my ( $option_html, $current_round ) = $select_html =~ m{
(<option\s+value="\d+"(?:\s+ selected="selected")?>(\d+)</option>)\Z
}isx;
my ($last_round_loaded) = $option_html =~ m{selected};
return ( $last_round_loaded, $current_round );
}
First off - don't use dequeue_nb(). This is a bad idea, because if a queue is temporarily empty, it'll return undef and your thread will exit.
Use instead dequeue and and end. dequeue will block, but once you end your queue, the while will exit.
You're also doing some decidedly odd things with your threads - I would suggest that you rarely want to detach a thread. You're just assuming your thread is going to complete before your program, which isn't a good plan.
Likewise this;
my $thread = threads->create(
sub { scrape_match( $thread_id, $queue, $queue_processed ) } )
->join;
You're spawning a thread, and then instantly joining it. And so that join call will... block waiting for your thread to exit. You don't need threads at all to do that...
You also scope your queues within your foreach loop. I don't think that's a good plan. I would suggest instead - scope them externally, and spawn a defined number of 'worker' threads (and one 'printing' thread).
And then just feed them through the queue mechanism. Otherwise you'll end up creating multiple queue instances, because they're lexically scoped.
And once you've finished queuing stuff, issue a $queue -> end which'll terminate the while loop.
You also don't need to give a thread a $thread_id because ... they already have one. Try: threads -> self -> tid(); instead.

Resources