I'm using threads in perl (5.12 ActiveState) to allow parallel and asyncronous writing on two different COM ports on Windows. This is how my code looks like:
#!/usr/bin/perl
use warnings;
use strict;
use Win32::SerialPort;
use threads;
my $ComPortObj = new Win32::SerialPort ("COM10") or die ("This is the bitter end...");
[... omit port settings ...]
my $ComPortObj2 = new Win32::SerialPort ("COM6") or die ("This is the bitter end...");
[... omit port settings ...]
my $s_read = "";
my $HangupThr = async
{
# printf("THREAD - Wait 3 seconds\n");
# sleep(3);
print("THREAD - write on COM10: AT\n");
$ComPortObj->write("AT\r") || die ("Unable to send command\n");
printf("THREAD - Wait 1 second\n");
sleep(1);
$s_read = $ComPortObj2->input;
# $s_read =~ s/\n/N/g;
# $s_read =~ s/\r/R/g;
print("THREAD - read from COM6: $s_read\n");
return 1;
};
$HangupThr->detach();
# printf("MAIN - Wait 4 seconds\n");
# sleep(4);
print("MAIN - write on COM6: AT\n");
$ComPortObj2->write("AT\r") || die ("Unable to send command\n");
printf("MAIN - Wait 1 second\n");
sleep(1);
$s_read = $ComPortObj->input;
# $s_read =~ s/\n/N/g;
# $s_read =~ s/\r/R/g;
print("MAIN - read from COM10: $s_read\n");
$ComPortObj->close();
$ComPortObj2->close();
What I get is an error when program exits. Complete output:
MAIN - write on COM6: AT
THREAD - write on COM10: AT
MAIN - Wait 1 second
THREAD - Wait 1 second
MAIN - read from COM10: AT
OK
THREAD - read from COM6: AT
OK
Error in PurgeComm at C:\userdata\Perl scripts\src\handler_error.pl line 0 thread 1
The operation completed successfully.
Error in GetCommTimeouts at C:\userdata\Perl scripts\src\handler_error.pl line 0 thread 1
Error Closing handle 184 for \\.\COM6
The handle is invalid.
Error closing Read Event handle 188 for \\.\COM6
The handle is invalid.
Error closing Write Event handle 192 for \\.\COM6
The handle is invalid.
Error in PurgeComm at C:\userdata\Perl scripts\src\handler_error.pl line 0 thread 1
The handle is invalid.
Error in GetCommTimeouts at C:\userdata\Perl scripts\src\handler_error.pl line 0 thread 1
Error Closing handle 144 for \\.\COM10
The handle is invalid.
Error closing Read Event handle 148 for \\.\COM10
The handle is invalid.
Error closing Write Event handle 180 for \\.\COM10
The handle is invalid.
This is related to serial port handlers purge, which I have no idea on how perl duplicates in threads. I've tried various close attempts in thread, main... without success. Furthermore I have to use the same ports both in main program and thread. Any suggestion to prevent these errors?
Many thanks!
You are dealing with Serial Ports and at any point only one process can have control on the serial ports(some terminal switches provide multiple login but thats not your case) In windows when one Process connects to COM it automatically disconnects other process. You can try this by trying to login to same COM port twice from the windows machine and the other port should disconnect which should lead to invalid handles you are seeing as error.
Other things you can try
Create Com Object Inside the thread, Use it and Destroy the object before accessing it on other thread
Related
I'm implementing features of an ssh server, so given a shell request I open a pty-tty pair.
A snippet:
import (
"github.com/creack/pty"
...
)
func attachPty(channel ssh.Channel, shell *exec.Cmd) {
mypty, err := pty.Start(shell)
go func() {
io.Copy(channel, mypty) // (1) ; could also be substituted with read() syscall, same problem
}
go func() {
io.Copy(mypty, channel) // (2) - this returns on channel exit with eof, so let's close mypty
if err := syscall.Close(int(mypty.Fd())); err != nil {
fmt.Printf("error closing fd") // no error is printed out, /proc/fd shows it's successfuly closed
}
}
}
Once the ssh channel gets closed, I close the pty. My expected behavior is that it should send SIGHUP to the shell.
If I comment out the (1) copy (src: mypty, dst: channel), it works!
However - when it's not commented out:
The (1) copy doesn't return, meaning the read syscall from mypty is still blocking, and doesn't return eof => master device doesn't get closed?
shell doesn't get SIGHUP
I'm not sure why if I comment out the (1) copy it works, maybe the kernel reference counts the reads?
My leads:
pty.read is actually dispatched to the tty, as said in:
pty master missing read function
Walkthrough of SIGHUP flow
pty_close in drivers/tty/pty.c, which calls tty_vhangup(tty->link);, see here
Linux Device Drivers, 3rd edition, PTY chapter
Go notes:
I close the fd directly, because otherwise using the usual os.File.close() doesn't actually close the fd for some reason, it stays open in /proc/<pid>/fd
substituting the (1) copy with a direct read syscall would lead to the same outcome
Thank you!
I've just started learning python, but I have problem with my code:
import pifacecad
# listener initialization
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
print('some prints...')
time.sleep(4)
print('and the end.')
blowMyMind() will be fired as many times as listener it tells to. That is okay.
My goal is to deactivate listener UNTIL blowMyMind ends. Pifacecad suggest Barrier() to achieve that, at least I think that it was here for that reason(correct me if I'm wrong).
Now it's working as many times as I activate listener event, but It's not like pushing function 99 times at once, but queues it and runs one by one.
With Barriers I think it should look like this:
# Barrier
global end_barrier
end_barrier = Barrier(1)
# listener initialization
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global end_barrier
test = end_barrier.wait()
print(test) # returns 0, which should not in about 5 seconds
print('some prints...')
time.sleep(4)
print('and the end.')
The funny part is when I change parties in Barrier initialization it is causing BrokenBarrierError at first listener event.
Actually I think that I completely misunderstood Barrier() I think the problem with it is that all listener events are in one thread instead of their own threads.
It's making me even more confused when I'm reading:
parties The number of threads required to pass the barrier.
from here: https://docs.python.org/3/library/threading.html
My conclusion: when initializing Barrier(X) it would be realeased when there will be X('or less', 'or more'?) number of threads. That sounds VERY stupid :D
I tried to make it that way with no luck:
# listener initialization
global busy
busy = 0
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global busy
if busy == 0:
busy = 1
print('some prints...')
time.sleep(4)
print('and the end.')
busy = 0
else:
return None
This question is two-fold.
1. So I need to run code for a socket server that's all defined and created in another.py, Clicking run on PyCharm works just fine, but if you exec() the file it just runs the bottom part of the code.
There are a few answers here but they are conflicting and for Python 2.
From what I can gather there are three ways:
- Execfile(), Which I think is Python 2 code.
- os.system() (But I've seen it be said that it's not correct to pass to the OS for this)
- And subprocess.Popen (unsure how to use this either)
I need this to run in the background, it is used to create threads for sockets for the recv portion of the overall program and listen on those ports so I can input commands to a router.
This is the complete code in question:
import sys
import socket
import threading
import time
QUIT = False
class ClientThread(threading.Thread): # Class that implements the client threads in this server
def __init__(self, client_sock): # Initialize the object, save the socket that this thread will use.
threading.Thread.__init__(self)
self.client = client_sock
def run(self): # Thread's main loop. Once this function returns, the thread is finished and dies.
global QUIT # Need to declare QUIT as global, since the method can change it
done = False
cmd = self.readline() # Read data from the socket and process it
while not done:
if 'quit' == cmd:
self.writeline('Ok, bye. Server shut down')
QUIT = True
done = True
elif 'bye' == cmd:
self.writeline('Ok, bye. Thread closed')
done = True
else:
self.writeline(self.name)
cmd = self.readline()
self.client.close() # Make sure socket is closed when we're done with it
return
def readline(self): # Helper function, read up to 1024 chars from the socket, and returns them as a string
result = self.client.recv(1024)
if result is not None: # All letters in lower case and without and end of line markers
result = result.strip().lower().decode('ascii')
return result
def writeline(self, text): # Helper func, writes the given string to the socket with and end of line marker at end
self.client.send(text.strip().encode("ascii") + b'\n')
class Server: # Server class. Opens up a socket and listens for incoming connections.
def __init__(self): # Every time a new connection arrives, new thread object is created and
self.sock = None # defers the processing of the connection to it
self.thread_list = []
def run(self): # Server main loop: Creates the server (incoming) socket, listens > creates thread to handle it
all_good = False
try_count = 0 # Attempt to open the socket
while not all_good:
if 3 < try_count: # Tried more than 3 times without success, maybe post is in use by another program
sys.exit(1)
try:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create the socket
port = 80
self.sock.bind(('127.0.0.1', port)) # Bind to the interface and port we want to listen on
self.sock.listen(5)
all_good = True
break
except socket.error:
print('Socket connection error... Waiting 10 seconds to retry.')
del self.sock
time.sleep(10)
try_count += 1
print('Server is listening for incoming connections.')
print('Try to connect through the command line with:')
print('telnet localhost 80')
print('and then type whatever you want.')
print()
print("typing 'bye' finishes the thread. but not the server",)
print("eg. you can quit telnet, run it again and get a different ",)
print("thread name")
print("typing 'quit' finishes the server")
try:
while not QUIT:
try:
self.sock.settimeout(0.500)
client = self.sock.accept()[0]
except socket.timeout:
time.sleep(1)
if QUIT:
print('Received quit command. Shutting down...')
break
continue
new_thread = ClientThread(client)
print('Incoming Connection. Started thread ',)
print(new_thread.getName())
self.thread_list.append(new_thread)
new_thread.start()
for thread in self.thread_list:
if not thread.isAlive():
self.thread_list.remove(thread)
thread.join()
except KeyboardInterrupt:
print('Ctrl+C pressed... Shutting Down')
except Exception as err:
print('Exception caught: %s\nClosing...' % err)
for thread in self.thread_list:
thread.join(1.0)
self.sock.close()
if "__main__" == __name__:
server = Server()
server.run()
print('Terminated')
Notes:
This is created in Python 3.4
I use Pycharm as my IDE.
One part of a whole.
2. So I'm creating a lightning detection system and this is how I expect it to be done:
- Listen to the port on the router forever
The above is done, but the issue with this is described in question 1.
- Pull numbers from a text file for sending text message
Completed this also.
- Send http get / post to port on the router
The issue with this is that i'm unsure how the router will act if I send this in binary form, I suspect it wont matter, the input commands for sending over GSM are specific. Some clarification may be needed at some point.
- Recieve reply from router and exception manage
- Listen for relay trip for alarm on severe or close strike warning.
- If tripped, send messages to phones in storage from text file
This would be the http get / post that's sent.
- Wait for reply from router to indicate messages have been sent, exception handle if it's not the case
- Go back to start
There are a few issues I'd like some background knowledge on that is proving hard to find via the old Google and here on the answers in stack.
How do I grab the receive data from the router from another process running in another file? I guess I can write into a text file and call that data but i'd rather not.
How to multi-process and which method to use.
How to send http get / post to socket on router, post needed occording to the router manual is as follows: e.g. "http://192.168.1.1/cgi-bin/sms_send?number=0037061212345&text=test"
Notes: Using Sockets, threading, sys and time on Python 3.4/Pycharm IDE.
Lightning detector used is LD-250 with RLO Relay attached.
RUT500 Teltonica router used.
Any direction/comments, errors spotted, anything i'm drastically missing would be greatly appreciated! Thank you very much in advance :D constructive criticism is greatly encouraged!
Okay so for the first part none of those suggested in the OP were my answer. Running the script as is from os.system(), exec() without declaring a new socket object just ran from __name__, this essentially just printed out "terminated", to get around this was simple. As everything was put into a classes already, all I had to do is create a new thread. This is how it was done:
import Socketthread2
new_thread = Socketthread2.Server() # Effectively declaring a new server class object.
new_thread.run()
This allowed the script to run from the beginning by initialising the code from the start in Socket, which is also a class of Clientthread, so that was also run too. Running this at the start of the parent program allowed this to run in the background, then continue with the new code in parent while the rest of the script was continuously active.
I'd like to have Parallel::ForkManager use a callback to get something back from a child process and then also restart it. Is that possible? The following is from the Parallel::ForkManager docs:
use strict;
use Parallel::ForkManager;
my $max_procs = 5;
my #names = qw( Fred Jim Lily Steve Jessica Bob Dave Christine Rico Sara );
# hash to resolve PID's back to child specific information
my $pm = new Parallel::ForkManager($max_procs);
# Setup a callback for when a child finishes up so we can
# get it's exit code
$pm->run_on_finish(
sub { my ($pid, $exit_code, $ident) = #_;
print "** $ident just got out of the pool ".
"with PID $pid and exit code: $exit_code\n";
}
);
$pm->run_on_start(
sub { my ($pid,$ident)=#_;
print "** $ident started, pid: $pid\n";
}
);
$pm->run_on_wait(
sub {
print "** Have to wait for one children ...\n"
},
0.5
);
foreach my $child ( 0 .. $#names ) {
my $pid = $pm->start($names[$child]) and next;
# This code is the child process
print "This is $names[$child], Child number $child\n";
sleep ( 2 * $child );
print "$names[$child], Child $child is about to get out...\n";
sleep 1;
$pm->finish($child); # pass an exit code to finish
#####here is where I'd like each child process to restart
}
So when $pm->finish happens, the callback confirms the "child" is "out of the pool." How can I both get the callback to fire and immediately put the child back in the pool as they come out, so that it runs forever?
I think you're misunderstanding what's happening. Under the covers, what Parallel::ForkManager is doing is calling a fork(). Two processes exist at this point, with only a single difference - different PIDs.
Your child process goes and runs some stuff, then exits, generating an exit status, which your parent then reaps.
Restarting the process... well, you just need to fork again and run your code.
Now, what you're doing - a foreach loop, that - foreach array element, forks and then the fork exits.
So really - all your need to do, is call $pm -> start again. How you figure out which one exited (and thus the child name) is more difficult though - your callback runs in the parent process, so data isn't being passed back aside from the exit status of your child. You'll need to figure out some sort of IPC to notify the necessary details.
Although - I'd point out #names isn't a hash, so treating it like one is going to have strange behaviour :).
Have you considered threading as an alternative? Threads are good for shared memory operations passing keyed subprocesses is something it's good at.
It seems using pipe in threads might cause the threads turn into zombie. In fact the commands in the pipe truned into zombie, not the threads. This does not happen very time which is annoying since it's hard to find out the real problem. How to deal with this issue? What causes these? Was it related to the pipe? How to avoid this?
The following is the codes that creates sample files.
#buildTest.pl
use strict;
use warnings;
sub generateChrs{
my ($outfile, $num, $range)=#_;
open OUTPUT, "|gzip>$outfile";
my #set=('A','T','C','G');
my $cnt=0;
while ($cnt<$num) {
# body...
my $pos=int(rand($range));
my $str = join '' => map $set[rand #set], 1 .. rand(200)+1;
print OUTPUT "$cnt\t$pos\t$str\n";
$cnt++
}
close OUTPUT;
}
sub new_chr{
my #chrs=1..22;
push #chrs,("X","Y","M", "Other");
return #chrs;
}
for my $chr (&new_chr){
generateChrs("$chr.gz",50000,100000)
}
The following codes will create zombie threads occasionally. Reason or trigger remains unknown.
#paralRM.pl
use strict;
use threads;
use Thread::Semaphore;
my $s = Thread::Semaphore->new(10);
sub rmDup{
my $reads_chr=$_[0];
print "remove duplication $reads_chr START TIME: ",`date`;
return 0 if(!-s $reads_chr);
my $dup_removed_file=$reads_chr . ".rm.gz";
$s->down();
open READCHR, "gunzip -c $reads_chr |sort -n -k2 |" or die "Error: cannot open $reads_chr";
open OUTPUT, "|sort -k4 -n|gzip>$dup_removed_file";
my ($last_id, $last_pos, $last_reads)=split('\t',<READCHR>);
chomp($last_reads);
my $last_length=length($last_reads);
my $removalCnts=0;
while (<READCHR>) {
chomp;
my #line=split('\t',$_);
my ($id, $pos, $reads)=#line;
my $cur_length=length($reads);
if($last_pos==$pos){
#may dup
if($cur_length>$last_length){
($last_id, $last_pos, $last_reads)=#line;
$last_length=$cur_length;
}
$removalCnts++;
next;
}else{
#not dup
}
print OUTPUT join("\t",$last_id, $last_pos, $last_reads, $last_length, "\n");
($last_id, $last_pos, $last_reads)=#line;
$last_length=$cur_length;
}
print OUTPUT join("\t",$last_id, $last_pos, $last_reads, $last_length, "\n");
close OUTPUT;
close READCHR;
$s->up();
print "remove duplication $reads_chr END TIME: ",`date`;
#unlink("$reads_chr")
return $removalCnts;
}
sub parallelRMdup{
my #chrs=#_;
my %jobs;
my #removedCnts;
my #processing;
foreach my $chr(#chrs){
while (${$s}<=0) {
# body...
sleep 10;
}
$jobs{$chr}=async {
return &rmDup("$chr.gz")
}
push #processing, $chr;
};
#wait for all threads finish
foreach my $chr(#processing){
push #removedCnts, $jobs{$chr}->join();
}
}
sub new_chr{
my #chrs=1..22;
push #chrs,("X","Y","M", "Other");
return #chrs;
}
¶llelRMdup(&new_chr);
As the comments on your originating post suggest - there isn't anything obviously wrong with your code here. What might be helpful to understand is what a zombie process is.
Specifically - it's a spawned process (by your open) which has exited, but the parent hasn't collected it's return code yet.
For short running code, that's not all that significant - when your main program exits, the zombies will 'reparent' to init which will clean them up automatically.
For longer running, you can use waitpid to clean them up and collect return codes.
Now in this specific case - I can't see a specific problem, but I would guess it's to do with how you're opening your filehandles. The downside of opening filehandles like you are, is that they're globally scoped - and that's just generally bad news when you're doing thready things.
I would imagine if you changed your open calls to:
my $pid = open ( my $exec_fh, "|-", "executable" );
And then called waitpid on that $pid following your close then your zombies would finish. Test the return from waitpid to get an idea of which of your execs has errored (if any), which should help you track down why.
Alternatively - set $SIG{CHLD} = "IGNORE"; which will mean you - effectively - tell your child processes to 'just go away immediately' - but you won't be able to get a return code from them if they die.