I have some code that spawns 5 threads of itself, at a time. I detach those threads, but have a shared variable $THREADCOUNT that I increment at the beginning of the subprocedure that is called by the thread call, and decrement at the end of the thread. When $THREADCOUNT equals 0, I spawn another 5 times.
The problem is, sometimes the thread exits unexpectedly and the $THREADCOUNT never makes it to 0, so the program stops. Is there someway to capture an exit like this and have $THREADCOUNT-- on unexpected exit?
Thanks so much. This is my first post so appologies if it's a little unclear.
Chris
Is the shared $THREADCOUNT variable really necessary? A call to threads->list(threads::running) will tell you whether any of your spawn are still running.
I can't figure out when anyone would ever want to use detach. I'd use something like
use threads;
my %workers;
sub start_worker {
my $thread = threads->create(#_);
$workers{$thread->tid} = $thread;
return $thread;
}
sub collect_finished_workers {
for my $thread (threads->list(threads::joinable)) {
$thread->join()
if delete($workers{$threads->tid}); # Don't assume we own all threads.
}
}
sub get_worker_count {
collect_finished_workers();
return 0+keys(%workers);
}
Note that this fixes the problem in your code where a thread isn't counted as started for a short while after it has started.
Related
I have a perl script, that goes through a queue of files and process files in different threads:
Below is a snippet after filling the files queue.
The problem is, sometimes when I leave it running for too long (12-24 hours), I come back to see the script hanging.
I have log files for each thread, I see that the last time stamp of activity is always the same for all of them. But the script does not exit, meaning that the threads do not give back the semaphore.
Also I am sure that the threads did not end properly, because I can still see the queue is full of files to be handled.
I have a log before and after calling the EXE, and the last trace is always the one after calling the EXE. Cannot find a rational explanation.
#initiate all threads
for( my $i = 0; $i < $max_thread; $i++ )
{
my $my_thread = threads->new( sub { start() } );
push( #Threads, $my_thread );
}
$semaphore->down($max_thread);
terminate();
sub start
{
$SIG{INT} = sub { thread_exit() };
while( (my $file = $file_queue->dequeue_nb) )
{
#This function calls an external EXE
processFile( $file );
}
#Thread ended
$semaphore->up();
}
If an uncaught exception is thrown in your thread, the semaphore will never be "upped". It looks like you are using the semaphore to wait for the thread to exit, which can be done more safely using $thread->join.
That's assuming you are correct about the thread having exited. It could also be that the thread has stalled (e.g. deadlocked) or entered an infinite loop.
use threads;
use threads::shared;
use Term::ReadKey;
sub input_worker {
local $SIG{'KILL'} = sub { threads->exit(0); return;};
while (1) {
if (defined(my $char = ReadKey 0, *STDIN)) {
print "$char";
}
}
return;
} ## end sub input_worker
my $in_thr = threads->create(\&input_worker);
sleep 5;
my $rc = $in_thr->kill('KILL')->join();
print "$rc\n";
This program wont exit by itself and just hangs.
It will only exit after pressing "enter"
How can i make it so that it will exit by itself after 'kill' is signaled
P.S. I dont want to use detach();
Mixing signals and threads is a bit of a challenge, simply because $thread->kill doesn't actually use real signals (since signals are sent to processes, not threads). Which is just as well, because if it did, SIGKILL can break things.
Even when talking about 'normal' signals - you will have slightly unexpected behaviour, because of perl's handling of them See: perlipc . It's not impossible to use, but you need to be aware of the caveats involved.
But the root of the problem is - that the signal is handled safely, which means perl waits until a suitable moment to process it. It will not do this during a 'read' operation, so the signal processing will get deferred.
I think likely what is happening here is your ReadKey is not a nonblocking operation like you think it is. With reference to the Term::ReadKey manpage
ReadKey MODE [, Filehandle]
Takes an integer argument, which can currently be one of the following values:
0 Perform a normal read using getc
-1 Perform a non-blocked read
>0 Perform a timed read
So what it does instead is - starts reading from STDIN, and blocks (ignoring signals) until you press enter. At which point it runs your signal handler.
Given what you're trying to do here - I would suggest you don't want to use signals at all, and instead - just use a shared variable.
Something like this:
use threads;
use threads::shared;
use Term::ReadKey;
my $done : shared;
sub input_worker {
while ( not $done ) {
if ( defined( my $char = ReadKey( -1, *STDIN ) ) ) {
print "$char";
}
}
return;
} ## end sub input_worker
my $in_thr = threads->create( \&input_worker );
sleep 10;
$done++;
my $rc = $in_thr->join();
print "$rc\n";
Will terminate after the timeout, and because it's doing nonblocking reads, it'll bail out on time, without input. You should note though - this thread is going to be 'spinning' looping rapidly waiting to see if input has been pressed - so it's not very CPU efficient. (a small delay in the cycle helps immensely there, say 0.1s).
I wrote a network packet listener program and I have 2 threads. Both runs forever but one of them sleeps 30 sec other sleeps 90 sec. In main function, I use sigaction function and after installed signal handler, I created these 2 threads. After creation of threads, main function calls pcaploop function, which is infinite loop. Basic structure of my program:
(I use pseudo syntax)
signalHandler()
only sets a flag (exitState = true)
thread1()
{
while 1
{
sleep 30 sec
check exit state, if so exit(0);
do smth;
}
}
thread2()
{
while 1
{
sleep 90 sec
check exit state, if so exit(0);
do smth;
}
}
main()
{
necassary snytax for sigaction ;
sigaction( SIGINT, &act, NULL );
sigaction( SIGUSR1, &act, NULL );
create thread1;
create thread2;
pcaploop(..., processPacket,...); // infinite loop, calls callback function (processPacket) everytime a packet comes.
join threads;
return 0;
}
processPacket()
{
check exitState, if true exit(0);
do smth;
}
And here is my question. When I press CTRL-C program does not terminate. If the program run less than 6-7 hours, when I press CTRL-C, program terminates. If the program run 1 night, at least 10 hours or more, I cannot terminate the program. Actually, signal handler is not called.
What could be the problem? Which thread does catch the signal?
Basically it would be better to remove all pseudo code you put in your example, and leave the minimum working code, what exactly you have.
From what I can see so far from your example, is that the error handling of sigaction's is missing.
Try to perform checks against errors in your code.
I am writing this for those who had faced with this problem. My problem was about synchronization of threads. After i got handle synchronization problem, the program now, can handle the signals. My advice is check the synchronization again and make sure that it works correctly.
I am sorry for late answer.
Edited :
I have also used sigaction for signal handling
and I have change my global bool variable whit this definition :
static volatile sig_atomic_t exitFlag = 0;
This flag has been used for checking whether the signal received or not.
What exactly does Perl do to threads that have completed its task? Does it let it idle or just kills it? I have a basic code structure below and I was wondering how to best optimize it.
use threads;
use Thread::Semaphore
my $s = Thread::Semaphore->new($maxThreads);
my #threads;
my $thread;
foreach my $tasktodo (#tasktodo) {
$s->down();
$thread = threads->new(\&doThis);
push #threads, $thread;
}
foreach my $thr (#threads) {
$thr->join();
}
sub doThis {
# blah blah
# completed, gonna let more threads run with $s->up()
$s->up();
}
In this case, once a thread completes, I want to free up resources for more threads to run. I'm worried about joining threads at the end of the loop. If in the whole program life cycle it will have 4 threads created, will #threads still have 4 threads in it when joining?
Lets say $maxThreads is 2, will it run 2 threads then when those 2 completes, it will be killed and run 2 more threads. At the end it will only join or wait for those 2 threads running?
EDIT: I also don't care for the return values of these threads, that's why I want to free up resources. Only reason I'm joining is I want all threads to complete before continuing with the script. Again, is this the best implementation?
The usual method for terminating a thread is to return EXPR from the entry point function with the appropriate return value(s).
The join function waits for this return value, and clean up the thread. So, in my opinion your code is fine.
Another way to exit a thread is this:
threads->exit(status);
Also, you can get a list of joinable threads with:
threads->list(threads::joinable);
I have a perl object with a few functions in it. Each functions is called once from the main program. I would like to run some of the functions in parallel to save time. I can't run all of them together since some functions depend on the results of previous functions.
I thought of something like this:
For each function keep a flag that is initialized to false and is set to true by the function when it ends (e.g. the last line in func1 would be $is_func1_done = 1).
Start each function with a loop that waits until all the flags of the functions it depends on are true. For example: if func1 depends on func2 and func3 then:
sub func1 {
while (!($is_func2_done && $is_func3_done)) {
# do nothing
}
# do work
}
Then I can start immediately a thread for each function, but the actual work of each function will start only when it's ready. Does this make sense? Do I need any locks here on the flags? Is using such while loops common? -- the term busy waiting comes to mind... maybe most of my CPU time will be spent on these whiles? Is there a more standard solution to this?
Does this make sense?
Yes - each task knows its preconditions, and waits for them to be met before executing. It's one of a number of valid designs, though you might find it difficult to scale as the number of tasks grow and their interdependencies grow more complex.
Do I need any locks here on the flags?
Yes. The flags need to be shared, so that one thread can manipulate them and another see it, and shared variables need to be lock()ed to be used safely.
Is using such while loops common? -- the term busy waiting comes to mind
Sadly yes, but Don't Do That, Please. Shared variables in perl can serve as condition variables through which threads can send notifications to one another:
sub func1 {
{
lock(%shared_state);
until ($shared_state{ "func2 done" } and $shared_state{ "func3 done" }) {
cond_wait(%shared_state);
}
}
# do work -- note that %shared_state is unlocked
# now tell others that we're done
lock(%shared_state);
$shared_state{ "func1 done" } = 1;
cond_broadcast(%shared_state);
# %shared_state will be unlocked, and broadcast delivered when we leave this scope
}
When you cond_wait, the shared variable is unlocked and your thread is put to sleep. No need to busy loop.
Is there a more standard solution to this?
$thr->join, as Sinan suggests, is an easy and natural way to wait for a specific thread to finish running. Thread::Semaphore can serve a similar but more complex function (and, helpfully, can be initialized to values less than zero). A common need to "wait for these 5 threads to finish something" can be achieved with a Thread::Barrier. TMTOWTDI.
You should use $thr->join to wait for a thread to finish.
For example:
#!/usr/bin/perl
use strict; use warnings;
use threads;
my #threads = map threads->create($_), qw( func1 func2 );
$_->join for #threads;
my $thr3 = threads->create('func3');
$thr3->join;
sub func1 {
for (1 .. 5) {
print "func1\n";
sleep 1 + rand 3;
}
return;
}
sub func2 {
for (1 .. 5) {
print "func2\n";
sleep 1 + rand 2;
}
return;
}
sub func3 {
print "Time to do some work\n";
}
I don't know if it is common to use such while loops: I would not.