I can not deal with Semapfor. As they run, so they passed on one stream? In fact, the variable $ n Randomnaya taken from 0 to 2. When the condition that $ n = $ it (# $ num - stream number), the message, then the flow $ num smokes. Others (# where $ n! = $ Num) should be skipped. The message "trade does not smoke." The situation is that it passes all the threads at once and where it is not necessary, he deduces that the flow $ num smokes (# in fact gives the correct number, but it does not fit the situation).
#! usr/bin/perl -w
use strict;
use warnings;
use 5.010;
use threads;
use threads::shared;
use Thread::Semaphore;
my $sem = Thread::Semaphore->new(1);
my $n = int rand(3);
say $n;
my $shr : shared = 1;
my $threads = 2;
my #threads;
for my $t ( 0 .. $threads ) {
push #threads, threads->create( \&smoke, $t );
}
# Дожидаемся окончания работы всех потоков
for my $t (#threads) { $t->join(); }
sub smoke {
my $num = shift;
$sem->down;
say "+Thread $num started";
sleep 1;
if ( $num = $n ) { sleep 2; say "Thread $num -- smoke"; }
say "-Thread $num done. \n";
if ( $num != $n ) {
say "-Thread $num dont smoke!. \n";
}
{ lock($shr); $shr++ }
$sem->up;
return;
sleep 1;
}
Semaphores are nothing more complicated than a shared counter.
When you call down() the counter is decreased - and it will block if it cannot. If you call up() the count is incremented. (And anything blocking will be released).
However, until that 'block' occurs, the threads can - and will - execute in an undefined order, that you should assume is random. (It isn't entirely, but relying on any particular sequence will create race conditions).
It's not complicated, but one of the things that can trip you up is buffering - print statements in threads may get buffered, so appear to arrive in strange order.
E.g.:
#! usr/bin/perl -w
use strict;
use warnings;
use 5.010;
use threads;
use threads::shared;
use Thread::Semaphore;
my $sem = Thread::Semaphore->new(1);
#turn off buffering
$|++;
sub wait_for_sem {
print threads -> self -> tid. ": waiting for semaphore\n";
$sem -> down();
print threads -> self -> tid. ": got semaphore\n";
sleep rand 3;
print threads -> self -> tid. ": releasing semaphore\n";
$sem -> up;
}
threads -> create ( \&wait_for_sem ) for 1..100;
foreach my $thr ( threads -> list ) {
print "Waiting for ".$thr->tid." to join\n";
$thr -> join;
print "Joined ".$thr->tid."\n";
}
I think buffering is what's going wrong with your code - try it with $|++.
Related
I write perl script to do a certain but I needed to mutlithread it, but I get this:
Perl exited with active threads:
2 running and unjoined
0 finished and unjoined
0 running and detached
Here is the code:
use Net::Ping;
use threads;
use Benchmark qw(:hireswallclock);
use threads::shared;
my $starttime = Benchmark->new;
my $finishtime;
my $timespent;
my $num_of_threads = 2;
my #threads = initThreads();
my $inFile = $ARGV[0] ;
open(IN , "<$inFile") or die "can not find $inFile" ;
my #output: shared = <IN>;
chomp (#output) ;
my $chunk_size = #output / 2;
print($chunk_size);
#############################
######## PROCEDURES ########
#############################
# Subroutine that intializes an array that will contain all our threads:
sub initThreads{
my #initThreads; # Our array
for(my $i=1; $i<=$num_of_threads; $i++){
push(#initThreads, $i);
}
return #initThreads;
}
sub doScript{
my $id = threads->tid();
print "//////////////////////////////////////////////////////////////////////////////////////////////Starting thread $id\n";
my ($start, $end, $output) = #_;
for my $i ($start .. $end) {
## some stuff done
sleep 1 if 0.2 > rand;
}
print "/////////////////////////////////////////////////////////////////////////////////////////////////////////////////Thread $id done!\n";
threads->exit();
}
########################
######## MAIN ########----------------------------------------------------------------
########################
for my $chunk(1 .. 2){
my $start = ($chunk - 1) * $chunk_size;
push #threads, threads->create(
\&doScript,
$start,
($start + $chunk_size - 1),
\#output,
);
print("finish");
}
# This tells the main program to keep running until all threads have finished.
foreach(#threads){
threads->join();
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime,$starttime);
print "\nDone!\nSpent ". timestr($timespent);
#print "\nProgram Done!\nPress Enter to exit\n";
$a = <>;
close (IN);
I even replaced the join block with this:
$_->join() for threads->list();
the error was solved but the script seemed not to do anything, threads started and terminated without doing anything.
Can anyone help me with this !
The reason you get that error is exactly as said on the tin - your code exited before the threads were closed. It can commonly happen when an exit or die is triggered early.
In particular I think your problem may lie within your reusing of #threads and initThreads().
The latter returns a list of numbers, not any threads. And then you push a couple more threads to the end of the list later, when you're doing a create.
That looks a lot like some sort of logic error to me.
But the major problem I think will be this:
foreach(#threads){
threads->join();
}
You're not actually joining a specific thread. What you probably want is:
foreach my $thr ( #threads ) {
$thr -> join();
}
At least, you would, if you hadn't manually populated #threads with [1,2] thanks to my #threads = initThreads();
Let's say we have a script which open a file, then read it line by line and print the line to the terminal. We have a sigle thread and a multithread version.
The problem is than the resulting output of both scripts is almost the same, but not exactly. In the multithread versions there are about ten lines which missed the first 2 chars. I mean, if the real line is something line "Stackoverflow rocks", I obtain "ackoverflow rocks".
I think that this is related to some race condition since if I adjust the parameters to create a lot of little workers, I get more faults than If I use less and bigger workers.
The single thread is like this:
$file = "some/file.txt";
open (INPUT, $file) or die "Error: $!\n";
while ($line = <STDIN>) {
print $line;
}
The multithread version make's use of the thread queue and this implementation is based on the #ikegami approach:
use threads qw( async );
use Thread::Queue 3.01 qw( );
use constant NUM_WORKERS => 4;
use constant WORK_UNIT_SIZE => 100000;
sub worker {
my ($job) = #_;
for (#$job) {
print $_;
}
}
my $q = Thread::Queue->new();
async { while (defined( my $job = $q->dequeue() )) { worker($job); } }
for 1..NUM_WORKERS;
my $done = 0;
while (!$done) {
my #lines;
while (#lines < WORK_UNIT_SIZE) {
my $line = <>;
if (!defined($line)) {
$done = 1;
last;
}
push #lines, $line;
}
$q->enqueue(\#lines) if #lines;
}
$q->end();
$_->join for threads->list;
I tried your program and got similar (wrong) results. Instead of Thread::Semaphore I used lock from threads::shared around the print as it's simpler to use than T::S, i.e.:
use threads;
use threads::shared;
...
my $mtx : shared;
sub worker
{
my ($job) = #_;
for (#$job) {
lock($mtx); # (b)locks
print $_;
# autom. unlocked here
}
}
...
The global variable $mtx serves as a mutex. Its value doesn't matter, even undef (like here) is ok.
The call to lock blocks and returns only if no other threads currently holds the lock on that variable.
It automatically unlocks (and thus makes lock return) when it goes out of scope. In this sample that happens
after every single iteration of the for loop; there's no need for an extra {…} block.
Now we have syncronized the print calls…
But this didn't work either, because print does buffered I/O (well, only O). So I forced unbuffered output:
use threads;
use threads::shared;
...
my $mtx : shared;
$| = 1; # force unbuffered output
sub worker
{
# as above
}
...
and then it worked. To my surprise I could then remove the lock and it still worked. Perhaps by accident. Note that your script will run significantly slower without buffering.
My conclusion is: you're suffering from buffering.
I am using Thread::Queue to push an array onto a queue and process each element of it using threads. Below is a simplified version of my program to demonstrate what is happening.
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
use Thread::Queue;
# Define queue
my $QUEUE :shared = new Thread::Queue();
# Define values
my #values = qw(string1 string2 string3);
# Enqueue values
$QUEUE->enqueue(#values);
# Get thread limit
my $QUEUE_SIZE = $QUEUE->pending();
my $thread_limit = $QUEUE_SIZE;
# Create threads
for my $i (1 .. $thread_limit) {
my $thread = threads->create(\&work);
}
# Join threads
my $i = 0;
for my $thread (threads->list()) {
$thread->join();
}
print "COMPLETE\n";
# Thread work function
sub work {
while (my $value = $QUEUE->dequeue()) {
print "VALUE: $value\n";
sleep(5);
print "Finished sleeping\n";
}
print "Got out of loop\n";
}
When I run this code I get the following output and then it just hangs forever:
VALUE: string1
VALUE: string2
VALUE: string3
Finished sleeping
Finished sleeping
Finished sleeping
Once the queue reaches its end, the while loop should break and the script should continue but it doesn't appear to ever get out of the loop.
Why is this getting stuck?
Since you never call $QUEUE->end(), your threads are blocking on dequeue() waiting for more entries to appear.
So, ensure you do call $QUEUE->end() after the last call to enqueue, or before joining the threads.
I'm running a perl script consisting of 30 threads to run a subroutine. For each thread, I'm supplying 100 data. In the subroutine, after the code does what its supposed to, I'm storing the output in a csv file. However, I find that on execution, the csv file has some data overlapped. For example, in the csv file, I'm storing name, age, gender, country this way-
print OUTPUT $name.",".$age.",".$gender.",".$country.",4\n";
The csv file should have outputs as such-
Randy,35,M,USA,4
Tina,76,F,UK,4
etc.
However, in the csv file, I see that some columns has overlapped or has been entered haphazardly in this way-
Randy,35,M,USA,4
TinaMike,76,UK
23,F,4
Is it because some threads are executing at the same time? What could I do to avoid this? I'm using the print statement only after I'm getting the data. Any suggestions?
4 is the group id which will remain constant.
Below is the code snippet:
#!/usr/bin/perl
use DBI;
use strict;
use warnings;
use threads;
use threads::shared;
my $host = "1.1.1.1";
my $database = "somedb";
my $user = "someuser";
my $pw = "somepwd";
my #threads;
open(PUT,">/tmp/file1.csv") || die "can not open file";
open(OUTPUT,">/tmp/file2.csv") || die "can not open file";
my $dbh = DBI->connect("DBI:mysql:$database;host=$host", $user, $pw ,) || die "Could not connect to database: $DBI::errstr";
$dbh->{'mysql_auto_reconnect'} = 1;
my $sql = qq{
//some sql to get a primary keys
};
my $sth = $dbh->prepare($sql);
$sth->execute();
while(my #request = $sth->fetchrow_array())
{
#get other columns and print to file1.csv
print PUT $net.",".$sub.",4\n";
$i++; #this has been declared before
}
for ( my $count = 1; $count <= 30; $count++) {
my $t = threads->new(\&sub1, $count);
push(#threads,$t);
}
foreach (#threads) {
my $num = $_->join;
print "done with $num\n";
}
sub sub1 {
my $num = shift;
//calculated start_num and end_num based on an internal logic
for(my $x=$start_num; $x<=$end_num; $x++){
print OUTPUT $name.",".$age.",".$gender.",".$country.",4\n";
$j++; #this has been declared before
}
sleep(1);
return $num;
}
I have problem in the file2 which has the OUTPUT handler
You are multithreading and printing to a file from multiple threads. This will always end badly - print is not an 'atomic' operation, so different prints can interrupt each other.
What you need to do is serialize your output such that this cannot happen. The simplest way is to use a lock or a semaphore:
my $print_lock : shared;
{
lock $print_lock;
print OUTPUT $stuff,"\n";
}
when the 'lock' drifts out of scope, it'll be released.
Alternatively, have a separate thread that 'does' file IO, and use Thread::Queue to feed lines to it. Depends somewhat on whether you need any ordering/processing of the contents of 'OUTPUT'.
Something like:
use Thread::Queue;
my $output_q = Thread::Queue -> new();
sub output_thread {
open ( my $output_fh, ">", "output_filename.csv" ) or die $!;
while ( my $output_line = $output_q -> dequeue() ) {
print {$output_fh} $output_line,"\n";
}
close ( $output_fh );
sub doing_stuff_thread {
$output_q -> enqueue ( "something to output" ); #\n added by sub!
}
my $output_thread = threads -> create ( \&output_thread );
my $doing_stuff_thread = threads -> create ( \&doing_stuff_thread );
#wait for doing_stuff to finish - closing the queue will cause output_thread to flush/exit.
$doing_stuff_thread -> join();
$output_q -> end;
$output_thread -> join();
Open the File handle globally, then try using flock on the file handle as demonstrated:
sub log_write {
my $line = shift;
flock(OUTPUT, LOCK_EX) or die "can't lock: $!";
seek(OUTPUT, 0, SEEK_END) or die "can't fast forward: $!";
print OUTPUT $line;
flock(OUTPUT, LOCK_UN) or die "can't unlock: $!";
}
Other example:
perlfaq5 - I still don't get locking. I just want to increment the number in the file. How can I do this?
While I see how to have Perl trap Ctrl-C (sigint) in bash; I'm getting lost at why does it fail with threads; I'm trying the following script:
#!/usr/bin/env perl
use threads;
use threads::shared; # for shared variables
my $cnt :shared = 0;
sub counter() {
while (1) {
$cnt++;
print "thread: $cnt \n";
sleep 1;
}
}
sub finisher{
### Thread exit! ...
print "IIII";
threads->exit();
die;
};
# any of these will cause stop of reaction to Ctrl-C
$SIG{INT} = \&finisher;
$SIG{INT} = sub {print "EEE\n" ;} ;
$SIG{INT} = 'IGNORE';
# setting to DEFAULT, brings usual behavior back
#~ $SIG{INT} = 'DEFAULT';
my $mthr = threads->create(\&counter);
$mthr->join();
... and as soon as the SIGINT handler is set to anything else than the default (where Ctrl-C causes exit), it basically causes for the script to stop reacting on Ctrl-C any longer:
$ ./test.pl
thread: 1
^Cthread: 2
^C^Cthread: 3
^C^C^C^Cthread: 4
thread: 5
thread: 6
thread: 7
thread: 8
Terminated
... and I have to sudo killall perl in order to terminate the script.
There is a bit on threads and Ctrl-C in these links:
Sending sig INT ( control C ) to threads - Dev Shed
SOLVED: Is this a bug of perl threads ? - Perl Monks
How do we capture CTRL ^ C - Perl Monks
Is the signal handler supposed to work like this?
... but I cannot say if it conclusively answers whether "capturing" Ctrl-C under perl in bash is definitely impossible?
Thanks in advance for any answers,
Cheers!
Ok, I think I got it (but I'm leaving the previous entry (below) for reference ...)
The trick turns out to be that, from the main SIGINT handler, one must signal the thread via kill - AND then thread also needs to have a separate SIGINT handler (from the first link in OP); AND instead of just join(), one needs to use the code in the answer by #ikegami:
#!/usr/bin/env perl
use threads;
use threads::shared; # for shared variables
my $cnt :shared = 0;
my $toexit :shared = 0;
sub counter() {
$SIG{'INT'} = sub { print "Thread exit\n"; threads->exit(); };
my $lexit = 0;
while (not($lexit)) {
{ lock($toexit);
$lexit = $toexit;
}
$cnt++;
print "thread: $cnt \n";
sleep 1;
}
print "out\n";
}
my $mthr;
sub finisher{
{ lock($toexit);
$toexit = 1;
}
$mthr->kill('INT');
};
$SIG{INT} = \&finisher;
$mthr = threads->create(\&counter);
print "prejoin\n";
#~ $mthr->join();
while (threads->list()) {
my #joinable = threads->list(threads::joinable);
if (#joinable) {
$_->join for #joinable;
} else {
sleep(0.050);
}
}
print "postjoin\n";
I may be overkilling it with the $toexit there, but at least now this is the result:
$ ./test.pl
prejoin
thread: 1
thread: 2
thread: 3
^CThread exit
postjoin
Many thanks to all for the solution :)
Cheers!
Thanks to the suggestion by #mob for PERL_SIGNALS to unsafe (note, Perl 5.14 does not allow "internal" setting of $ENV{'PERL_SIGNALS'}), I'm getting somewhere - now Ctrl-C is detected - but it either terminates with a segfault, or with error:
#!/usr/bin/env perl
use threads;
use threads::shared; # for shared variables
my $cnt :shared = 0;
my $toexit :shared = 0;
sub counter() {
my $lexit = 0;
while (not($lexit)) {
{ lock($toexit);
$lexit = $toexit;
}
$cnt++;
print "thread: $cnt \n";
sleep 1;
}
print "out\n";
#~ threads->detach(); # Thread 1 terminated abnormally: Cannot detach a joined thread
#~ exit;
}
my $mthr;
# [http://code.activestate.com/lists/perl5-porters/164923/ [perl #92246] Perl 5.14 does not allow "internal" setting of $ENV ...]
sub finisher{
### Thread exit! ...
#~ print "IIII";
# anything here results with: Perl exited with active threads:
#~ threads->exit();
#~ threads->join();
#~ $mthr->exit();
#~ $mthr->join();
#~ $mthr->detach();
#~ $mthr->kill();
#~ threads->exit() if threads->can('exit'); # Thread friendly
#~ die;
{ lock($toexit);
$toexit = 1;
}
#~ threads->join(); #
};
# any of these will cause stop of reaction to Ctrl-C
$SIG{INT} = \&finisher;
#~ $SIG{INT} = sub {print "EEE\n" ; die; } ;
#~ $SIG{INT} = 'IGNORE';
# setting to DEFAULT, brings usual behavior back
#~ $SIG{INT} = 'DEFAULT';
$mthr = threads->create(\&counter);
print "prejoin\n";
$mthr->join();
print "postjoin\n";
With the comments as above, that code react with:
$ PERL_SIGNALS="unsafe" ./testloop06.pl
prejoin
thread: 1
thread: 2
thread: 3
^Cthread: 4
out
Segmentation fault
Result is the same if I add the following that uses Perl::Signals::Unsafe:
$mthr = threads->create(\&counter);
UNSAFE_SIGNALS {
$mthr->join();
};
Almost there, hopefully someone can chime in ... :)
Signal handlers are only called between Perl opcodes. Your code is blocked in $mthr->join();, so it never gets to handle the signal.
Possible solution:
use Time::HiRes qw( sleep );
# Interruptable << $_->join() for threads->list; >>
while (threads->list()) {
my #joinable = threads->list(threads::joinable);
if (#joinable) {
$_->join for #joinable;
} else {
sleep(0.050);
}
}