Perl TK Gui freezes - multithreading

I have a perl tk application where in i create many objects and update the perl tk gui display with information in objects.I need to add large number of jobs(say 30k) in the tree in the gui.If i add all jobs in one go,the gui freezes.
Below is the code snippet:
sub Importjobs
{
#================= start creation of objects=============================
my JobList $self = shift;
my $exportedJobList = shift;
# third parameter whether to clear the list
$self->clear () unless shift;
my $noOfProcsToBeAdded = shift || 3000;
my $cellCollection = Tasks::CellCollection::instance ();
my $calcActionsPathHash = $cellCollection->caPathCAHash ();
my $collectionCellNames = $cellCollection->allCellNames ();
my #importedJobs = ();
# if the given job list is empty, add import job list to it
push #{$self->_importJobList()}, #$exportedJobList;
$exportedJobList = [];
# do not import new jobs if the previous jobs are still being created
foreach my $taskGenJob(#{$self->getTaskGenJobObjs()}) {
goto FINISH if TaskGenJobState::CREATE == $taskGenJob->state();
}
# now get each job and add it into the imported jobs till the noOfJobs exceeds $noOfJobsToBeAdded
while(my $jobDescription = shift #{$self->_importJobList()}) {
my $taskInstantiation = $jobDescription->{'taskInstantiation'};
my $caPath = $taskInstantiation->{'calcActionPath'};
my $errMsgPrefix = 'Error importing ' . join ('-', $task, $command, $method, $caPath);
my #calcActionList;
if(defined $caPath) {
my $calcAction = $calcActionsPathHash->{ $caPath };
unless($calcAction) {
my $errMsg = $errMsgPrefix . ": the calcAction is not defined within the current CellCollection : " . $caPath;
$logger4Perl -> error ($errMsg);
next;
}
push #calcActionList, $calcAction;
} else {
my #mList;
if(not defined $method) {
push #mList, #{$task->getMethods(cellCollection => $cellCollection, command => $command)};
$method = join(' ', #mList);
} elsif($method eq $task_desc::default_method) {
#mList = ($task_desc::default_method);
} else {
#mList = sort (grep { $_ } split(/\s+|__/, $method));
}
foreach my $m (#mList) {
push(#calcActionList, #{$cellCollection->findCalcActions($task, $command, $m)});
}
}
foreach my $calcAction(#calcActionList) {
my TaskGenJob $job = TaskGenJob->new ();
$logger4Perl->info ("Adding $caPath");
push (#importedJobs, $job);
my $noOfProcsBeingAdded = $job->calculateNoOfJobExecObjs();
$noOfProcsToBeAdded -= $noOfProcsBeingAdded;
}
last if 1 > $noOfProcsToBeAdded;
}
#================= End creation of objects=============================
#Below function updates the GUI display
$self->addJobs (\#importedJobs);
#================= Mechanism which am using so that GUI will be active after certain time limit=============================
FINISH:
if(#{$self->_importJobList()}) {
$self->parentDlg()->parentWnd()->after(60000,
sub {
$GuiTasksAppl::mainDlg->Disable();
$self->importJobList([], 'noclear', 200);
$GuiTasksAppl::mainDlg->Enable();
});
}
}
Currently the way am doing it is to add say 3000 jobs using $noOfProcsToBeAdded variable and stay idle for some time and repeat the process after some time.During this idle process,there is different process which processes the jobs in GUI.
can someone propose a better approach than this ???
Expecting ideas on threading ,shared memory.

First, if the GUI freezes (and never unfreezes) during your large 30k update then you might have found a Tk bug since that shouldn't happen. However, if its merely unresponsive for a period of time, then it make sense to mitigate the delay.
In the past, i've used either Tk::repeat() or Tk::after() to drive my UI update routine. The user interface doesn't typically need to be updated at a high rate, so every few hundred milliseconds can be a reasonable delay. The determining factor is largely determined by how responsive of an interface you need. Then during the job import step: append references to a list for the UI update routine and then periodically call $MW->update(). The update routine doesn't necessarily need to process the full list during each call but you don't want the processing to get too far behind.
I'd also recommend some visual indicator to identify that the update is still in-progress.
If ImportJobs is computationally expensive, obviously one could perform multi-process / multi-threading tricks to exploit multiple processors on the system. But that'll add a bit of complexity and testing effort.

Related

SOAP::Lite - clients using version 1.1 and 1.2 threaded in mod_perl

I have several SOAP::Lite clients running under mod_perl in Apache hhtpd.
Some of them use 1.1 soap-servers and some of them use 1.2 servers. So I have code like:
# In client 1:
my $soap1 = SOAP::Lite->soapversion("1.1");
$result1 = $soap1->method1();
# In client 2:
my $soap2 = SOAP::Lite->soapversion("1.2");
$result2 = $soap2->method2();
This works in stand-alone clients, but when I run the code under mod_perl, I seem to get stung by that the soapversion
method has side-effects:
# From SOAP::Lite.pm
sub soapversion {
my $self = shift;
my $version = shift or return $SOAP::Constants::SOAP_VERSION;
($version) = grep {
$SOAP::Constants::SOAP_VERSIONS{$_}->{NS_ENV} eq $version
} keys %SOAP::Constants::SOAP_VERSIONS
unless exists $SOAP::Constants::SOAP_VERSIONS{$version};
die qq!$SOAP::Constants::WRONG_VERSION Supported versions:\n#{[
join "\n", map {" $_ ($SOAP::Constants::SOAP_VERSIONS{$_}->{NS_ENV})"} keys %SOAP::Constants::SOAP_VERSIONS
]}\n!
unless defined($version) && defined(my $def = $SOAP::Constants::SOAP_VERSIONS{$version});
foreach (keys %$def) {
eval "\$SOAP::Constants::$_ = '$SOAP::Constants::SOAP_VERSIONS{$version}->{$_}'";
}
$SOAP::Constants::SOAP_VERSION = $version;
return $self;
}
This is what I believe happens:
Basically, the soapversion call rededefines essential constants in $SOAP::Constants. And since this is mod_perl, the $SOAP::Constants are global and shared between every server-thread (I believe. Please correct me if I'm wrong). This leads to a race-condition: Most of the times, the codelines gets executed more-or-less in the sequence seen above. But once in at while (actually about 2% of the calls) the execution sequence is:
Thread1: my $soap1 = SOAP::Lite->soapversion("1.1");
Thread2: my $soap2 = SOAP::Lite->soapversion("1.2");
Thread1: $result1 = $soap1->method1();
Thread2: $result2 = $soap2->method2();
And so, the $soap1->method1() gets called with $SOAP::Constants set as befitting version 1.2 - causing several namespace to be wrong, notably:
xmlns:soapenc="http://www.w3.org/2003/05/soap-encoding"
Which is wrong for 1.1 - who prefers:
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
If I could somehow make $SOAP::Constants localized to a serverthread or something like that, I guess things would be fine. But any solution will be appreciated.
Run Apache with the prefork model instead of the threading model (mpm_prefork_module instead of mpm_event_module or mpm_worker_module), so that each Apache child will have its own Perl interpreter, hence its own set of constants.
Otherwise have a look on the modperl documentation regarding the PerlOptions directive, specifically the clone and/or parent value. But stop using threads seem simpler to me, threads and Perl were never friends.

Perl Device::SerialPort

Looking for right way to detect one keyword during board boot up message.
After keyword detected, send Enter key after one second.
Kernel is Linux.
# Serial port inisialisation is finished here.
# Read boot message
($count, $result) = $ob->read(300); # at least 300 chars coming till keyword appear
if ($result =~ m/Booting_up/) {
print "send Enter ...\n";
sleep 1;
$ob->write("\r\n");
}
Thanks for hint
It appears that you are using Win32::SerialPort module, or perhaps Device::SerialPort which
provides an object-based user interface essentially identical to the one provided by the Win32::SerialPort module.
Its method read takes the number of bytes to read and returns the number read and writes them into the given string.
You may be "missing" the phrase because it's past the 300-mark, and your code doesn't read any further. Try to loop, getting a few bytes at a time and adding them up, thus building the string in small reads.
my bytes_in = 10; # length of pattern, but it does NOT ensure anything
my ($read, $result);
while (1)
{
my ($count, $read) = $ob->read( $bytes_in );
$result = $result . $read;
if ($result =~ m/Booting_up/) { # is it "Booting_up" or "Booting up" ?
print "send Enter ...\n";
sleep 1; # is this needed?
$ob->write("\r\n");
# last; # in case this is all you need to do
}
last if $count != $bytes_in; # done reading
}
I don't put the $ob->read statement in the loop condition since the documentation isn't crystal clear on how the method works. You may also be able to simply use
while ( my ($count, $read) = $ob->read( $bytes_in ) ) {
$result = $result . $read;
if ($result =~ m/Booting_up/s) {
# ...
}
last if $count != $bytes_in;
}
We read a small number of bytes at a time to prevent problems with either polling or blocking reads, brought up in comments by BenPen. See Configuration and capability methods.
You can first read those first 300 bytes that precede the pattern in one go and then start reading a few (or one) at a time, which would also lead to the quickest identification of the phrase.
This can be tweaked further but let's first see what it does as it stands (I cannot test).
Documentation also offers a few other methods which may be useful, in particular readline and streamline. As this is all rather low level there are yet other ways but if you got all else working perhaps this will be enough to complete it.
Perhaps rather index the string?
($count, $result) = $ob->read(300); # at least 300 chars coming till keyword appear
$substring = 'Booting_up';
if (index($result, $substring) != -1) {
print "send Enter ..\n";
sleep 1;
$ob->write("\r\n");
}

basic chat system on perl under linux

Im trying to write some basic chat system just to learn perl. Im trying to get the chatlog into a 1 file and print new message if it's appears in the chatlog.dat file, So i've wrote a function that does almost the same thing, but I have got some problems and don't know how to solve them.
So now I have 2 problems!
I could not understand how to keep checkFile function always active (like multiprocession) to continuously check for new messages
This problem occurs when I'm trying to write a new message that will be appended into the chatlog. The Interpreter waits for my input on the line my $newMessage = <STDIN>;, but, what if someone writes a new message? it will not be shown until he press enter... how to void that?
my ($sec,$min,$hour) = localtime();
while(1){
my $userMessage = <STDIN>;
last if $userMessage eq "::quit";
`echo "($hour:$min:$sec): $userMessage" >>chatlog.dat`;
}
sub checkFile{
my $lastMessage = "";
my $newMessage = "";
while (1) {
my $context = `cat chatlog.dat`;
split(/\n/, $context);
$newMessage = $_[$#_];
if ($newMessage ne $lastMessage) {
print $newMessage;
$lastMessage = $newMessage;
}
}
}
First:
don't use echo within a perl script. It's nasty to shell escape when you've got perfectly good IO routines.
using cat to read files is about as nasty as using 'echo'.
reading <STDIN> like that will be a blocking call - which means your script will pause.
but that's not as bad as it sounds, because otherwise you're running a 'busy wait' loop which'll repeatedy cat the file. This is a very bad idea.
You're assuming writing a file like that is an atomic operation, when it's not. You'll hit problems with doing that too.
What I would suggest you do it look at IO::Handle and also consider using flock to ensure you've got the file locked for IO. You may also wish to consider File::Tail instead.
I would actually suggest though, you want to consider a different mode of IPC - as 'file swapping' is quite inefficient. If you really want to use the filesystem for your IO, you might want to consider using a FIFO pipe - have each 'client' open it's own, and have a server reading and coalescing them.
Either way though - you'll either need to use IO::Select or perhaps multithreading, just to swap back and forth between reading and writing. http://perldoc.perl.org/IO/Select.html
Answering my own question
sub checkFile{
my $lastMessage = "";
my $newMessage = "";
my $userName = $_[0];
while (1) {
my $context = `cat chatlog.dat`;
split(/\n/, $context);
$newMessage = $_[$#_];
if ($newMessage ne $lastMessage) {
$newMessage =~ /^\(.+\):\((.+)\) (.+$)/;
if ($1 ne $userName) { print "[$1]: $2";}
$lastMessage = $newMessage;
}
}
}
my $userName = "Rocker";
my ($sec,$min,$hour) = localtime();
my $thr = threads -> create ( \&checkFile, $userName ); #Starting a thread to continuously check for the file update
while (1) {
my $userMessage = <STDIN>; #STDIN will not interfere file checking
last if $userMessage eq "::quit";
`echo "($hour:$min:$sec):($userName) $userMessage" >>chatlog.dat` if $userMessage =~ /\S+/;
}
$thr -> join();

Array getting destroyed when thread exits in perl

I was trying to use threading for parsing two different types of files. The subroutines share no data at all.
# Parse header files
$hdr_thrd = threads -> create(\&Parser::parse_header_file, $path);
# Parse input template files
$tmplt_thrd = threads -> create(\&TemplateParser::parse_template);
# Join the threads
$tmplt_thrd -> join();
$hdr_thrd -> join();
# This uses the data obtained from the above two threads
&Parser::parse_xml_template();
The problem comes when the parse_xml_template function tries to access an array #TemplateParser::array. The array has no data at this point but it is getting filled inside the parse_template function. Am I missing something?
You're trying to share data across threads without sharing it. You need to use :shared or share() on the variable.
You wouldn't have this problem at all if you were to avoid global vars as you should.
sub parse_template {
my #tmplt_result;
...
return \#tmplt_result;
}
my $hdr_thrd = threads->create(\&Parser::parse_header_file, $path);
my $tmplt_thrd = threads->create(\&TemplateParser::parse_template);
my $tmplt_result = $tmplt_thrd->join();
my $hdr_result = $hdr_thrd->join();
(Explicit sharing is not necessary to when returning a value via join.)
Of course, that needlessly creates two threads (for a total of three) when two would suffice. Instead, you could use:
sub parse_template {
my #tmplt_result;
...
return \#tmplt_result;
}
my $hdr_thrd = threads->create(\&Parser::parse_header_file, $path);
my $tmplt_result = TemplateParser::parse_template();
my $hdr_result = $hdr_thrd->join();

How to deal with multiple threads in perl which turn into zombie

It seems using pipe in threads might cause the threads turn into zombie. In fact the commands in the pipe truned into zombie, not the threads. This does not happen very time which is annoying since it's hard to find out the real problem. How to deal with this issue? What causes these? Was it related to the pipe? How to avoid this?
The following is the codes that creates sample files.
#buildTest.pl
use strict;
use warnings;
sub generateChrs{
my ($outfile, $num, $range)=#_;
open OUTPUT, "|gzip>$outfile";
my #set=('A','T','C','G');
my $cnt=0;
while ($cnt<$num) {
# body...
my $pos=int(rand($range));
my $str = join '' => map $set[rand #set], 1 .. rand(200)+1;
print OUTPUT "$cnt\t$pos\t$str\n";
$cnt++
}
close OUTPUT;
}
sub new_chr{
my #chrs=1..22;
push #chrs,("X","Y","M", "Other");
return #chrs;
}
for my $chr (&new_chr){
generateChrs("$chr.gz",50000,100000)
}
The following codes will create zombie threads occasionally. Reason or trigger remains unknown.
#paralRM.pl
use strict;
use threads;
use Thread::Semaphore;
my $s = Thread::Semaphore->new(10);
sub rmDup{
my $reads_chr=$_[0];
print "remove duplication $reads_chr START TIME: ",`date`;
return 0 if(!-s $reads_chr);
my $dup_removed_file=$reads_chr . ".rm.gz";
$s->down();
open READCHR, "gunzip -c $reads_chr |sort -n -k2 |" or die "Error: cannot open $reads_chr";
open OUTPUT, "|sort -k4 -n|gzip>$dup_removed_file";
my ($last_id, $last_pos, $last_reads)=split('\t',<READCHR>);
chomp($last_reads);
my $last_length=length($last_reads);
my $removalCnts=0;
while (<READCHR>) {
chomp;
my #line=split('\t',$_);
my ($id, $pos, $reads)=#line;
my $cur_length=length($reads);
if($last_pos==$pos){
#may dup
if($cur_length>$last_length){
($last_id, $last_pos, $last_reads)=#line;
$last_length=$cur_length;
}
$removalCnts++;
next;
}else{
#not dup
}
print OUTPUT join("\t",$last_id, $last_pos, $last_reads, $last_length, "\n");
($last_id, $last_pos, $last_reads)=#line;
$last_length=$cur_length;
}
print OUTPUT join("\t",$last_id, $last_pos, $last_reads, $last_length, "\n");
close OUTPUT;
close READCHR;
$s->up();
print "remove duplication $reads_chr END TIME: ",`date`;
#unlink("$reads_chr")
return $removalCnts;
}
sub parallelRMdup{
my #chrs=#_;
my %jobs;
my #removedCnts;
my #processing;
foreach my $chr(#chrs){
while (${$s}<=0) {
# body...
sleep 10;
}
$jobs{$chr}=async {
return &rmDup("$chr.gz")
}
push #processing, $chr;
};
#wait for all threads finish
foreach my $chr(#processing){
push #removedCnts, $jobs{$chr}->join();
}
}
sub new_chr{
my #chrs=1..22;
push #chrs,("X","Y","M", "Other");
return #chrs;
}
&parallelRMdup(&new_chr);
As the comments on your originating post suggest - there isn't anything obviously wrong with your code here. What might be helpful to understand is what a zombie process is.
Specifically - it's a spawned process (by your open) which has exited, but the parent hasn't collected it's return code yet.
For short running code, that's not all that significant - when your main program exits, the zombies will 'reparent' to init which will clean them up automatically.
For longer running, you can use waitpid to clean them up and collect return codes.
Now in this specific case - I can't see a specific problem, but I would guess it's to do with how you're opening your filehandles. The downside of opening filehandles like you are, is that they're globally scoped - and that's just generally bad news when you're doing thready things.
I would imagine if you changed your open calls to:
my $pid = open ( my $exec_fh, "|-", "executable" );
And then called waitpid on that $pid following your close then your zombies would finish. Test the return from waitpid to get an idea of which of your execs has errored (if any), which should help you track down why.
Alternatively - set $SIG{CHLD} = "IGNORE"; which will mean you - effectively - tell your child processes to 'just go away immediately' - but you won't be able to get a return code from them if they die.

Resources