perl multithreading perl exited with active threads - multithreading

I write perl script to do a certain but I needed to mutlithread it, but I get this:
Perl exited with active threads:
2 running and unjoined
0 finished and unjoined
0 running and detached
Here is the code:
use Net::Ping;
use threads;
use Benchmark qw(:hireswallclock);
use threads::shared;
my $starttime = Benchmark->new;
my $finishtime;
my $timespent;
my $num_of_threads = 2;
my #threads = initThreads();
my $inFile = $ARGV[0] ;
open(IN , "<$inFile") or die "can not find $inFile" ;
my #output: shared = <IN>;
chomp (#output) ;
my $chunk_size = #output / 2;
print($chunk_size);
#############################
######## PROCEDURES ########
#############################
# Subroutine that intializes an array that will contain all our threads:
sub initThreads{
my #initThreads; # Our array
for(my $i=1; $i<=$num_of_threads; $i++){
push(#initThreads, $i);
}
return #initThreads;
}
sub doScript{
my $id = threads->tid();
print "//////////////////////////////////////////////////////////////////////////////////////////////Starting thread $id\n";
my ($start, $end, $output) = #_;
for my $i ($start .. $end) {
## some stuff done
sleep 1 if 0.2 > rand;
}
print "/////////////////////////////////////////////////////////////////////////////////////////////////////////////////Thread $id done!\n";
threads->exit();
}
########################
######## MAIN ########----------------------------------------------------------------
########################
for my $chunk(1 .. 2){
my $start = ($chunk - 1) * $chunk_size;
push #threads, threads->create(
\&doScript,
$start,
($start + $chunk_size - 1),
\#output,
);
print("finish");
}
# This tells the main program to keep running until all threads have finished.
foreach(#threads){
threads->join();
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime,$starttime);
print "\nDone!\nSpent ". timestr($timespent);
#print "\nProgram Done!\nPress Enter to exit\n";
$a = <>;
close (IN);
I even replaced the join block with this:
$_->join() for threads->list();
the error was solved but the script seemed not to do anything, threads started and terminated without doing anything.
Can anyone help me with this !

The reason you get that error is exactly as said on the tin - your code exited before the threads were closed. It can commonly happen when an exit or die is triggered early.
In particular I think your problem may lie within your reusing of #threads and initThreads().
The latter returns a list of numbers, not any threads. And then you push a couple more threads to the end of the list later, when you're doing a create.
That looks a lot like some sort of logic error to me.
But the major problem I think will be this:
foreach(#threads){
threads->join();
}
You're not actually joining a specific thread. What you probably want is:
foreach my $thr ( #threads ) {
$thr -> join();
}
At least, you would, if you hadn't manually populated #threads with [1,2] thanks to my #threads = initThreads();

Related

Perl multithreading slower when memory usage getting high

Hi all~ I has written a very simple code in Perl using multithreading. The codes are as follows.
#!/bin/perl
use strict;
use threads;
use Benchmark qw(:hireswallclock);
my $starttime;
my $finishtime;
my $timespent;
my $num_of_threads = 1;
my $total_size = 10000000;
my $chunk_size = int($total_size / $num_of_threads);
if($total_size % $num_of_threads){
$chunk_size++;
}
my #threads = ();
$starttime = Benchmark->new;
for(my $i = 0; $i < $num_of_threads; $i++) {
my $thread = threads->new(\&search);
push (#threads, $thread);
}
foreach my $thread (#threads) {
$thread->join();
}
my $finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "$num_of_threads threads used in ".timestr($timespent)."\nDone!\n";
sub search{
my $i = 0;
while($i < $chunk_size){
$i++;
}
return 1;
}
This piece of codes works fine as when increasing the number of threads, it will run faster.
However, when adding additional lines in the middle, which will create an array in big size, the code will run more slowly when adding more threads. The codes with the additional lines are shown below.
#!/bin/perl
use strict;
use threads;
use Benchmark qw(:hireswallclock);
my $starttime;
my $finishtime;
my $timespent;
my $num_of_threads = 1;
my $total_size = 10000000;
my $chunk_size = int($total_size / $num_of_threads);
if($total_size % $num_of_threads){
$chunk_size++;
}
##########Additional codes##########
print "Preparing data...\n";
$starttime = Benchmark->new;
my #array = ();
for(my $i = 0; $i < $total_size; $i++){
my $rn = rand();
push(#array, $rn);
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "Used ".timestr($timespent)."\n";
######################################
my #threads = ();
$starttime = Benchmark->new;
for(my $i = 0; $i < $num_of_threads; $i++) {
my $thread = threads->new(\&search);
push (#threads, $thread);
}
foreach my $thread (#threads) {
$thread->join();
}
my $finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "$num_of_threads threads used in ".timestr($timespent)."\nDone!\n";
sub search{
my $i = 0;
while($i < $chunk_size){
$i++;
}
return 1;
}
I am so confused regarding such a behaviour in Perl's multithreading. Does anyone has any idea what may go wrong here?
Thanks!
You have to remember that when using ithreads - interpreter threads - the entire Perl interpreter, including code and memory, is cloned into the new thread. So the more data to be cloned, the longer it takes. There are ways to control what is cloned; take a look at the threads perldoc.
You should do as little as possible and not even load many modules until after you have spawned your threads.
If you do have a lot of data that will be used by all the threads, make it shared with threads::shared. Then to share a data structure use shared_clone(). You cannot simply share() anything, other than a simple variable. That shared variable can only contain plain scalars or other shared references.
If you are going to consume or pump that data, make it a queue instead with the Thread::Queue module. It automatically shares the values and takes care of the locking. After spawning your pool of worker threads, control them with Thread::Semaphore. That way they won't terminate before you've given them anything to do. You can also prevent race conditions.
https://metacpan.org/pod/threads::shared
HTH
Thank you all for pointing me to the relative directions! I have learned and tried different things, inlucding how to use shared and queue, which should be able to solve the problem. So I revised the script as follows:
#!/bin/perl
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Benchmark qw(:hireswallclock);
my $starttime;
my $finishtime;
my $timespent;
my $num_of_threads = shift #ARGV;
my $total_size = 100000;
######Initiation of a 2D queue######
print "Preparing queue...\n";
$starttime = Benchmark->new;
my $queue = Thread::Queue->new();
for(my $i = 0; $i < $total_size; $i++){
my $rn1 = rand();
my $rn2 = rand();
my #interval :shared = sort($rn1, $rn2);
$queue->enqueue(\#interval);
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "Used ".timestr($timespent)."\n";
#####################################
$starttime = Benchmark->new;
my $queue_copy = $queue; #Copy the 2D queue so that the original queue can be kept\
for(my $i = 0; $i < $num_of_threads; $i++) {
my $thread = threads->create(\&search, $queue_copy);
}
foreach my $thread (threads->list()) {
$thread->join();
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "$num_of_threads threads used in ".timestr($timespent)."\nDone!\n";
#####################################
sub search{
my $temp_queue = $_[0];
while(my $temp_interval = $temp_queue->dequeue_nb()){
#Do something
}
return 1;
}
What I was trying to do was , first, to make a queue of arrays, with each containing two numbers. A copy of the queue was made as I want to preserve the original queue when going through it. Then the copied queue was gone through using multithreads. However, I still found out that it ran slower when more threads were added, which I have no clue why.

Threaded code exits before all tasks are complete

I am trying to take a portion of an existing script and have it run multiple nmap scans simultaneously to increase the speed of the script.
I initially tried using fork, but it was suggested to me that I should be using threads instead as I am doing this on a Windows box. I modified a code snippet I found online and it partially works.
I am using a list of 23 IP addresses. I have been able to open 10 threads and scan the first 10 addresses, but then the code exits. Ideally, the code would open a new thread each time one exits so that there are always 10 threads running, until it reaches the remainder, in this case there would be three. Then only 3 threads would be open.
This entire code needs to be run inside a subroutine that I have in my original sequential code. I am using ping instead of the nmap command to test.
#!/usr/bin/Perl
use strict;
use threads;
my $i = 0;
my #lines;
# Define the number of threads
my $num_of_threads = 10;
# use the initThreads subroutine to create an array of threads.
my #threads = initThreads();
my #files = glob( "./ping.txt" ) or die "Can't open CMS HostInventory$!"; # Open the CMS Host Inventory csv files for parsing
foreach my $file ( #files ) {
open (CONFIG, '<', $file) or die "Can't ip360_DNS File$!";
#lines = <CONFIG>;
chomp (#lines);
}
# Loop through the array:
foreach ( #threads ) {
# Tell each thread to perform our 'doOperation()' subroutine.
$_ = threads->create(\&doOperation);
}
# This tells the main program to keep running until all threads have finished.
foreach ( #threads ) {
$_->join();
}
print "\nProgram Done!\nPress Enter to exit";
$a = <>;
####################### SUBROUTINES ############################
sub initThreads{
my #initThreads;
for ( my $i = 1; $i <= $num_of_threads; $i++ ) {
push(#initThreads, $i);
}
return #initThreads;
}
sub doOperation{
# Get the thread id. Allows each thread to be identified.
my $id = threads->tid();
my $ip = ($id - 1);
system("ping $lines[$ip] >> ./input/$lines[$ip].txt");
print "Thread $id done!\n";
# Exit the thread
threads->exit();
}

Stop a thread when another thread stops in Perl

Hi I am trying to stop or kill the thread $t2 when $t1 stops. Here is the code:
#!/usr/local/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
use Time::HiRes qw( sleep );
print "Starting main program\n";
my #threads;
my $t1 = threads->new(\&c1);
my $t2 = threads->new(\&progress);
#my $t2 = threads->new(\&c2);
push(#threads,$t1);
push(#threads,$t2);
my $ret1 = $t1->join();
print "Thread returned: $ret1\n";
$t2->join();
if (!($t1->is_running()))
{
$t2->exit();
}
print "End of main program\n";
sub c1
{
print "Inside thread 1\n";
sleep 5;
return 245;
}
sub c2
{
print "Inside thread 2\n";
}
sub progress
{
$| = 1; # Disable buffering on STDOUT.
my $BACKSPACE = chr(0x08);
my #seq = qw( | / - \ );
for (;;) {
print $seq[0];
push #seq, shift #seq;
sleep 0.200;
print $BACKSPACE;
}
print "$BACKSPACE $BACKSPACE";
}
But thread $t2 continues running. Please help me out with this.
How to kill thread $t2.
I am confused with join(), detach(), exit()
You can't call exit on thread instance $t2->exit() and module warns you about that,
perl -Mthreads -we '$_->exit(3) and $_->join for async {sleep 4}'
Usage: threads->exit(status) at -e line 1
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
However, you can send a signal to thread (check signal list)
# Send a signal to a thread
$thr->kill('SIGUSR1');
You can use a shared variable:
use threads::shared;
my $done : shared;
$done = 0;
my $t1 = threads->new( sub { c1(); $done = 1;} );
And in the progress function:
# ...
for (;!$done;) {
# ....

Control+C handling in multithread perl script

I'm new in perl and I have a problem with handling ^C in my perl script. When I'm trying to continue script execution after recieving ^C during the sleep I have only output before $FLAG = 2; and nothing after:
# perl test.pl
sleeping...
^Cawaiking... =
#
instead of:
# perl test.pl
sleeping...
awaiking... ====
some..
#
It seams that ^C is killing the progres bar thread and after it dies no actions but print could be performed in the main thread. Could anyone help me with this problem?
$SIG{INT} = 'IGNORE';
our $FLAG : shared = 1;
...
sub call1{
$FLAG = 1;
my $pBar = threads->new(\&progressBarInit);
$pBar->detach;
print "sleeping...\n";
sleep 5;
print "awaiking...\n";
$FLAG = 2;
print "some..\n";
return #result;
}
call1();
sub progressBarInit{
my $max = 50;
my $counter = 1;
while($FLAG == 1){
progressBar( $counter, $max, 50, '=' );
$counter++;
if($counter > $max){$counter=1;}
sleep 1;
}
}
sub progressBar {
my ( $counter, $max, $width, $char ) = #_;
local $| = 1;
printf " %-${width}s\r", $char x (($width-1)*$counter/$max);
}
I think the problem is that you setup the signal handler in the parent.
According to this: http://perldoc.perl.org/threads.html
Signal handlers need to be set up in the threads for the signals they are expected to act upon. Here's an example for cancelling a thread:
Instead of using a flag you could use signals to communicate:
sub progressBarInit {
# Thread 'cancellation' signal handler
$SIG{'KILL'} = sub { threads->exit(); };
$SIG{INT} = 'IGNORE';
...
}
...
# Signal the thread to terminate, and then detach
# it so that it will get cleaned up automatically
my $pBar = threads->new(\&progressBarInit);
print "sleeping...\n";
sleep 5;
print "awaiking...\n";
$pBar->kill('KILL')->detach();

Perl trapping Ctrl-C with threads in bash

While I see how to have Perl trap Ctrl-C (sigint) in bash; I'm getting lost at why does it fail with threads; I'm trying the following script:
#!/usr/bin/env perl
use threads;
use threads::shared; # for shared variables
my $cnt :shared = 0;
sub counter() {
while (1) {
$cnt++;
print "thread: $cnt \n";
sleep 1;
}
}
sub finisher{
### Thread exit! ...
print "IIII";
threads->exit();
die;
};
# any of these will cause stop of reaction to Ctrl-C
$SIG{INT} = \&finisher;
$SIG{INT} = sub {print "EEE\n" ;} ;
$SIG{INT} = 'IGNORE';
# setting to DEFAULT, brings usual behavior back
#~ $SIG{INT} = 'DEFAULT';
my $mthr = threads->create(\&counter);
$mthr->join();
... and as soon as the SIGINT handler is set to anything else than the default (where Ctrl-C causes exit), it basically causes for the script to stop reacting on Ctrl-C any longer:
$ ./test.pl
thread: 1
^Cthread: 2
^C^Cthread: 3
^C^C^C^Cthread: 4
thread: 5
thread: 6
thread: 7
thread: 8
Terminated
... and I have to sudo killall perl in order to terminate the script.
There is a bit on threads and Ctrl-C in these links:
Sending sig INT ( control C ) to threads - Dev Shed
SOLVED: Is this a bug of perl threads ? - Perl Monks
How do we capture CTRL ^ C - Perl Monks
Is the signal handler supposed to work like this?
... but I cannot say if it conclusively answers whether "capturing" Ctrl-C under perl in bash is definitely impossible?
Thanks in advance for any answers,
Cheers!
Ok, I think I got it (but I'm leaving the previous entry (below) for reference ...)
The trick turns out to be that, from the main SIGINT handler, one must signal the thread via kill - AND then thread also needs to have a separate SIGINT handler (from the first link in OP); AND instead of just join(), one needs to use the code in the answer by #ikegami:
#!/usr/bin/env perl
use threads;
use threads::shared; # for shared variables
my $cnt :shared = 0;
my $toexit :shared = 0;
sub counter() {
$SIG{'INT'} = sub { print "Thread exit\n"; threads->exit(); };
my $lexit = 0;
while (not($lexit)) {
{ lock($toexit);
$lexit = $toexit;
}
$cnt++;
print "thread: $cnt \n";
sleep 1;
}
print "out\n";
}
my $mthr;
sub finisher{
{ lock($toexit);
$toexit = 1;
}
$mthr->kill('INT');
};
$SIG{INT} = \&finisher;
$mthr = threads->create(\&counter);
print "prejoin\n";
#~ $mthr->join();
while (threads->list()) {
my #joinable = threads->list(threads::joinable);
if (#joinable) {
$_->join for #joinable;
} else {
sleep(0.050);
}
}
print "postjoin\n";
I may be overkilling it with the $toexit there, but at least now this is the result:
$ ./test.pl
prejoin
thread: 1
thread: 2
thread: 3
^CThread exit
postjoin
Many thanks to all for the solution :)
Cheers!
Thanks to the suggestion by #mob for PERL_SIGNALS to unsafe (note, Perl 5.14 does not allow "internal" setting of $ENV{'PERL_SIGNALS'}), I'm getting somewhere - now Ctrl-C is detected - but it either terminates with a segfault, or with error:
#!/usr/bin/env perl
use threads;
use threads::shared; # for shared variables
my $cnt :shared = 0;
my $toexit :shared = 0;
sub counter() {
my $lexit = 0;
while (not($lexit)) {
{ lock($toexit);
$lexit = $toexit;
}
$cnt++;
print "thread: $cnt \n";
sleep 1;
}
print "out\n";
#~ threads->detach(); # Thread 1 terminated abnormally: Cannot detach a joined thread
#~ exit;
}
my $mthr;
# [http://code.activestate.com/lists/perl5-porters/164923/ [perl #92246] Perl 5.14 does not allow "internal" setting of $ENV ...]
sub finisher{
### Thread exit! ...
#~ print "IIII";
# anything here results with: Perl exited with active threads:
#~ threads->exit();
#~ threads->join();
#~ $mthr->exit();
#~ $mthr->join();
#~ $mthr->detach();
#~ $mthr->kill();
#~ threads->exit() if threads->can('exit'); # Thread friendly
#~ die;
{ lock($toexit);
$toexit = 1;
}
#~ threads->join(); #
};
# any of these will cause stop of reaction to Ctrl-C
$SIG{INT} = \&finisher;
#~ $SIG{INT} = sub {print "EEE\n" ; die; } ;
#~ $SIG{INT} = 'IGNORE';
# setting to DEFAULT, brings usual behavior back
#~ $SIG{INT} = 'DEFAULT';
$mthr = threads->create(\&counter);
print "prejoin\n";
$mthr->join();
print "postjoin\n";
With the comments as above, that code react with:
$ PERL_SIGNALS="unsafe" ./testloop06.pl
prejoin
thread: 1
thread: 2
thread: 3
^Cthread: 4
out
Segmentation fault
Result is the same if I add the following that uses Perl::Signals::Unsafe:
$mthr = threads->create(\&counter);
UNSAFE_SIGNALS {
$mthr->join();
};
Almost there, hopefully someone can chime in ... :)
Signal handlers are only called between Perl opcodes. Your code is blocked in $mthr->join();, so it never gets to handle the signal.
Possible solution:
use Time::HiRes qw( sleep );
# Interruptable << $_->join() for threads->list; >>
while (threads->list()) {
my #joinable = threads->list(threads::joinable);
if (#joinable) {
$_->join for #joinable;
} else {
sleep(0.050);
}
}

Resources