Problem
Hey, recently when I'm using pyqt6's QProcess, I try to use os.kill() to kill a QProcess's instance. (The reason why I want to use os.kill() instead of QProcess().kill() is that I want to send a CTRL_C_EVENT signal when killing the process.) Even though with using correct pid (acquired by calling QProcess().processId()), it seems that a signal would be sent to all processes unexpectedly.
Code
Here's my code:
from PyQt6.QtCore import QProcess
import os
import time
import signal
process_a = QProcess()
process_a.start("python", ['./test.py'])
pid_a = process_a.processId()
print(f"pid_a = {pid_a}")
process_b = QProcess()
process_b.start("python", ['./test.py'])
pid_b = process_b.processId()
print(f"pid_b = {pid_b}")
os.kill(pid_a, signal.CTRL_C_EVENT)
try:
time.sleep(1)
except KeyboardInterrupt:
print("A KeyboardInterrupt should not be caught here.")
process_a.waitForFinished()
process_b.waitForFinished()
print(f"process_a: {process_a.readAll().data().decode('gbk')}")
print(f"process_b: {process_b.readAll().data().decode('gbk')}")
and ./test.py is simple:
import time
time.sleep(3)
print("Done")
What I'm expecting
pid_a = 19956
pid_b = 28468
process_a:
process_b: Done
What I've got
pid_a = 28040
pid_b = 23708
A KeyboardInterrupt should not be caught here.
process_a:
process_b:
Discussion
I don't know whether this is a bug or misusage. It seems that signal.CTRL_C_EVENT is sent to all processes. So, how do I kill one QProcess instance with signal CTRL_C_EVENT correctly?
Related
I need to read output from a child process as it's produced -- perhaps not on every write, but well before the process completes. I've tried solutions from the Python3 docs and SO questions here and here, but I still get nothing until the child terminates.
The application is for monitoring training of a deep learning model. I need to grab the test output (about 250 bytes for each iteration, at roughly 1-minute intervals) and watch for statistical failures.
I cannot change the training engine; for instance, I cannot insert stdout.flush() in the child process code.
I can reasonably wait for a dozen lines of output to accumulate; I was hopeful of a buffer-fill solving my problem.
Code: variations are commented out.
Parent
cmd = ["/usr/bin/python3", "zzz.py"]
# test_proc = subprocess.Popen(
test_proc = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
out_data = ""
print(time.time(), "START")
while not "QUIT" in str(out_data):
out_data = test_proc.stdout
# out_data, err_data = test_proc.communicate()
print(time.time(), "MAIN received", out_data)
Child (zzz.py)
from time import sleep
import sys
for _ in range(5):
print(_, "sleeping", "."*1000)
# sys.stdout.flush()
sleep(1)
print("QUIT this exercise")
Despite sending lines of 1000+ bytes, the buffer (tested elsewhere as 2kb; here, I've gone as high as 50kb) filling doesn't cause the parent to "see" the new text.
What am I missing to get this to work?
Update with regard to links, comments, and iBug's posted answer:
Popen instead of run fixed the blocking issue. Somehow I missed this in the documentation and my experiments with both.
universal_newline=True neatly changed the bytes return to string: easier to handle on the receiving end, although with interleaved empty lines (easy to detect and discard).
Setting bufsize to something tiny (e.g. 1) didn't affect anything; the parent still has to wait for the child to fill the stdout buffer, 8k in my case.
export PYTHONUNBUFFERED=1 before execution did fix the buffering problem. Thanks to wim for the link.
Unless someone comes up with a canonical, nifty solution that makes these obsolete, I'll accept iBug's answer tomorrow.
subprocess.run always spawns the child process, and blocks the thread until it exits.
The only option for you is to use p = subprocess.Popen(...) and read lines with s = p.stdout.readline() or p.stdout.__iter__() (see below).
This code works for me, if the child process flushes stdout after printing a line (see below for extended note).
cmd = ["/usr/bin/python3", "zzz.py"]
test_proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
out_data = ""
print(time.time(), "START")
while not "QUIT" in str(out_data):
out_data = test_proc.stdout.readline()
print(time.time(), "MAIN received", out_data)
test_proc.communicate() # shut it down
See my terminal log (dots removed from zzz.py):
ibug#ubuntu:~/t $ python3 p.py
1546450821.9174328 START
1546450821.9793346 MAIN received b'0 sleeping \n'
1546450822.987753 MAIN received b'1 sleeping \n'
1546450823.993136 MAIN received b'2 sleeping \n'
1546450824.997726 MAIN received b'3 sleeping \n'
1546450825.9975247 MAIN received b'4 sleeping \n'
1546450827.0094354 MAIN received b'QUIT this exercise\n'
You can also do it with a for loop:
for out_data in test_proc.stdout:
if "QUIT" in str(out_data):
break
print(time.time(), "MAIN received", out_data)
If you cannot modify the child process, unbuffer (from package expect - install with APT or YUM) may help. This is my working parent code without changing the child code.
test_proc = subprocess.Popen(
["unbuffer"] + cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
I have the following simplyfied code. It listens to the D-Bus and does something when a new job is created. For that to work I need to start the GLib.MainLoop().run(), as it was presented by multiple examples I found.
While doing that, I want the program to continuously listen to the IPC bus and do something when a message is received. But obviously that doesn't work since my program is stuck at GLib.MainLoop().run().
How to implement something that let's me listen to the D-Bus and to the IPC at the same time?
#!/usr/bin/env python3.4
import asgi_ipc as asgi
from gi.repository import GLib
from pydbus import SystemBus
from systemd.daemon import notify as sd_notify
def main():
bus = SystemBus()
systemd = bus.get(".systemd1")
systemd.onJobNew = do_something_with_job()
channel_layer = asgi.IPCChannelLayer(prefix="endercp")
# Notify systemd this unit is ready
sd_notify("READY=1")
GLib.MainLoop().run()
while True:
message = channel_layer.receive(["endercp"])
if message is not (None, None):
do_something_with_message(message)
if __name__ == "__main__":
# Notify systemd this unit is starting
sd_notify("STARTING=1")
main()
# Notify systemd this unit is stopping
sd_notify("STOPPING=1")
Since IPCChannelLayer.receive() does not block, you can run it in an idle callback. Try this:
callback_id = GLib.idle_add(GLib.PRIORITY_DEFAULT_IDLE, poll_channel, data=channel_layer)
GLib.MainLoop().run()
GLib.idle_remove_by_data(channel_layer)
# ...
def poll_channel(channel_layer):
message = channel_layer.receive(["endercp"])
if message is not (None, None):
do_something_with_message(message)
return GLib.SOURCE_CONTINUE
This question is two-fold.
1. So I need to run code for a socket server that's all defined and created in another.py, Clicking run on PyCharm works just fine, but if you exec() the file it just runs the bottom part of the code.
There are a few answers here but they are conflicting and for Python 2.
From what I can gather there are three ways:
- Execfile(), Which I think is Python 2 code.
- os.system() (But I've seen it be said that it's not correct to pass to the OS for this)
- And subprocess.Popen (unsure how to use this either)
I need this to run in the background, it is used to create threads for sockets for the recv portion of the overall program and listen on those ports so I can input commands to a router.
This is the complete code in question:
import sys
import socket
import threading
import time
QUIT = False
class ClientThread(threading.Thread): # Class that implements the client threads in this server
def __init__(self, client_sock): # Initialize the object, save the socket that this thread will use.
threading.Thread.__init__(self)
self.client = client_sock
def run(self): # Thread's main loop. Once this function returns, the thread is finished and dies.
global QUIT # Need to declare QUIT as global, since the method can change it
done = False
cmd = self.readline() # Read data from the socket and process it
while not done:
if 'quit' == cmd:
self.writeline('Ok, bye. Server shut down')
QUIT = True
done = True
elif 'bye' == cmd:
self.writeline('Ok, bye. Thread closed')
done = True
else:
self.writeline(self.name)
cmd = self.readline()
self.client.close() # Make sure socket is closed when we're done with it
return
def readline(self): # Helper function, read up to 1024 chars from the socket, and returns them as a string
result = self.client.recv(1024)
if result is not None: # All letters in lower case and without and end of line markers
result = result.strip().lower().decode('ascii')
return result
def writeline(self, text): # Helper func, writes the given string to the socket with and end of line marker at end
self.client.send(text.strip().encode("ascii") + b'\n')
class Server: # Server class. Opens up a socket and listens for incoming connections.
def __init__(self): # Every time a new connection arrives, new thread object is created and
self.sock = None # defers the processing of the connection to it
self.thread_list = []
def run(self): # Server main loop: Creates the server (incoming) socket, listens > creates thread to handle it
all_good = False
try_count = 0 # Attempt to open the socket
while not all_good:
if 3 < try_count: # Tried more than 3 times without success, maybe post is in use by another program
sys.exit(1)
try:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create the socket
port = 80
self.sock.bind(('127.0.0.1', port)) # Bind to the interface and port we want to listen on
self.sock.listen(5)
all_good = True
break
except socket.error:
print('Socket connection error... Waiting 10 seconds to retry.')
del self.sock
time.sleep(10)
try_count += 1
print('Server is listening for incoming connections.')
print('Try to connect through the command line with:')
print('telnet localhost 80')
print('and then type whatever you want.')
print()
print("typing 'bye' finishes the thread. but not the server",)
print("eg. you can quit telnet, run it again and get a different ",)
print("thread name")
print("typing 'quit' finishes the server")
try:
while not QUIT:
try:
self.sock.settimeout(0.500)
client = self.sock.accept()[0]
except socket.timeout:
time.sleep(1)
if QUIT:
print('Received quit command. Shutting down...')
break
continue
new_thread = ClientThread(client)
print('Incoming Connection. Started thread ',)
print(new_thread.getName())
self.thread_list.append(new_thread)
new_thread.start()
for thread in self.thread_list:
if not thread.isAlive():
self.thread_list.remove(thread)
thread.join()
except KeyboardInterrupt:
print('Ctrl+C pressed... Shutting Down')
except Exception as err:
print('Exception caught: %s\nClosing...' % err)
for thread in self.thread_list:
thread.join(1.0)
self.sock.close()
if "__main__" == __name__:
server = Server()
server.run()
print('Terminated')
Notes:
This is created in Python 3.4
I use Pycharm as my IDE.
One part of a whole.
2. So I'm creating a lightning detection system and this is how I expect it to be done:
- Listen to the port on the router forever
The above is done, but the issue with this is described in question 1.
- Pull numbers from a text file for sending text message
Completed this also.
- Send http get / post to port on the router
The issue with this is that i'm unsure how the router will act if I send this in binary form, I suspect it wont matter, the input commands for sending over GSM are specific. Some clarification may be needed at some point.
- Recieve reply from router and exception manage
- Listen for relay trip for alarm on severe or close strike warning.
- If tripped, send messages to phones in storage from text file
This would be the http get / post that's sent.
- Wait for reply from router to indicate messages have been sent, exception handle if it's not the case
- Go back to start
There are a few issues I'd like some background knowledge on that is proving hard to find via the old Google and here on the answers in stack.
How do I grab the receive data from the router from another process running in another file? I guess I can write into a text file and call that data but i'd rather not.
How to multi-process and which method to use.
How to send http get / post to socket on router, post needed occording to the router manual is as follows: e.g. "http://192.168.1.1/cgi-bin/sms_send?number=0037061212345&text=test"
Notes: Using Sockets, threading, sys and time on Python 3.4/Pycharm IDE.
Lightning detector used is LD-250 with RLO Relay attached.
RUT500 Teltonica router used.
Any direction/comments, errors spotted, anything i'm drastically missing would be greatly appreciated! Thank you very much in advance :D constructive criticism is greatly encouraged!
Okay so for the first part none of those suggested in the OP were my answer. Running the script as is from os.system(), exec() without declaring a new socket object just ran from __name__, this essentially just printed out "terminated", to get around this was simple. As everything was put into a classes already, all I had to do is create a new thread. This is how it was done:
import Socketthread2
new_thread = Socketthread2.Server() # Effectively declaring a new server class object.
new_thread.run()
This allowed the script to run from the beginning by initialising the code from the start in Socket, which is also a class of Clientthread, so that was also run too. Running this at the start of the parent program allowed this to run in the background, then continue with the new code in parent while the rest of the script was continuously active.
I want to use Qt's QThreadPool, but it seems to be hanging my application if the workers in the queue do not finish before calling QApplication.quit(). Can anyone tell me if i'm doing something wrong in the reduced testcase below?
import logging
log = logging.getLogger(__name__)
import sys
from PySide import QtCore
import time
class SomeWork(QtCore.QRunnable):
def __init__(self, sleepTime=1):
super(SomeWork, self).__init__()
self.sleepTime = sleepTime
def run(self):
time.sleep(self.sleepTime)
print "work", QtCore.QThread.currentThreadId()
def _test(argv):
logging.basicConfig(level=logging.NOTSET)
app = QtCore.QCoreApplication(argv)
pool = QtCore.QThreadPool.globalInstance()
TASK_COUNT = int(argv[1]) if len(argv) > 1 else 1
mainThread = QtCore.QThread.currentThreadId()
print "Main thread: %s"%(mainThread)
print "Max thread count: %s"%(pool.maxThreadCount())
print "Work count: %s"%(TASK_COUNT)
for i in range(TASK_COUNT):
pool.start(SomeWork(1))
def boom():
print "boom(); calling app.quit()"
app.quit()
QtCore.QTimer.singleShot(2000, boom)
#import signal
#signal.signal(signal.SIGINT, signal.SIG_DFL)
return app.exec_()
if __name__ == '__main__':
sys.exit(_test(sys.argv))
To be clear, this is the output I get:
(env)root#localhost:# python test_pool.py 1
Main thread: 3074382624
Max thread count: 1
Work count: 1
work 3061717872
boom(); calling app.quit()
(env)root#workshop:/home/workshop/workshop/workshop# python test_pool.py 20
Main thread: 3074513696
Max thread count: 1
Work count: 20
work 3060783984
boom(); calling app.quit()
And it hangs forever on the second command, but not the first.
Thanks for any help you may have.
EDIT:
To be clear, I expect that if app.quit() is called while threads are in the thread queue, they do not run. Already running threads should run to completion. Then, the application should close.
This example fails on a Windows machine as well
This example works on the same Windows machine, but using PyQt4
Adding this to _test() just before the exec() fixes the issue, although all the threads run:
def waitForThreads():
print "Waiting for thread pool"
pool.waitForDone()
app.aboutToQuit.connect(waitForThreads)
I'm writing a pyqt program utilising the Qwizard. From one screen I am running a thread and starting a command line program via Popen. However I'm getting the following warning message:
QObject::connect: Cannot queue arguments of type 'QTextCursor'
(Make sure 'QTextCursor' is registered using qRegisterMetaType().)
The program usually continues but crashes sporadically immediately after this point, leading me to feel that it is related.
As the code of the program is too much to attach, here are the relevant snippets:
class Fee(QThread):
def __init__(self, parent=None, options=None):
QThread.__init__(self, parent)
#set the given options
if options:
self.__options = options
def copy(self, devicename, outputfilename):
self.__device = devicename
self.__outputfile = outputfilename
self.start()
def run(self):
cmd = self.__getCommandLine(self.options, self.__device, self.__outputfile)
logging.info("Running Fee with command %s" % (cmd))
#start the process
output = ""
process = Popen(shlex.split(cmd), stdout= PIPE, stderr= STDOUT)
stderr = process.communicate()[1]
class FeeManager(QtCore.QObject):
def copyAndVerify(self):
self.__fee = Fee(self, self.options)
self.connect(self.__fee, QtCore.SIGNAL("finished()"), self._setCopyFinished)
self.connect(self.__fee, QtCore.SIGNAL("progressUpdated(QString, int, QString)"), self._setCopyProgress)
devicename = self.device.blockdevice
self.__image_outputfilename = "output_file.img")
self.__fee.copy(devicename, self.__image_outputfilename)
class FeeWizardPage(QtGui.QWizardPage):
def initializePage(self):
#already earlier in another wizard page is the following: self.feemanager = FeeManager(self, info)
self.connect(self.wizard().feemanager, QtCore.SIGNAL("copyProgressUpdated(QString, int, QString)"), self.updateCopyProgress)
self.connect(self.wizard().feemanager, QtCore.SIGNAL("verifyProgressUpdated(QString, int, QString)"), self.updateVerifyProgress)
self.connect(self.wizard().feemanager, QtCore.SIGNAL("finished(PyQt_PyObject)"), self.setDone)
self.wizard().feemanager.copyAndVerify()
What am I doing wrong? How can I avoid this message and hopefully bring some stability to the program. I've searched the internet and this forum, and while I've tried a number of suggestions for others, none have worked for this problem. Even when I comment out all signals and connects, I still get the same warning message.
Can someone help?
Thanks a lot.