I've got a long running builder that renders videos. Sometimes during the build I'll notice that it's incorrect and want to kill it, and this requires sending an HTTP request to the render server. Is there some signal or hook in the scons system that I can use to determine when scons is shutting down so that I can send a message to the server?
You can do something like this: (original code from SCons manpage)
Put this in your SConstruct/SConscript
import atexit
def shutdown_my_server():
" do your stuff here "
pass
atexit.register(shutdown_my_server)
Related
I will try to give you some context before presenting the issue I am facing.
I have a component called Actuator which relies on the module pymodbus. When I tested this component, I did it in the easiest possible way using a modbus TCP server based on pymodbus. Basically, I run the server as a python script (which is exactly this one: https://pymodbus.readthedocs.io/en/latest/source/example/synchronous_server.html) in a new shell and my application, which includes the Actuator, in another shell. Everything works like a charm.
What I want to do now, is to write a unit test for the Actuator, using python unittest and pymodbus and try to automate what I was doing before. My idea was to run the modbus server as a subprocess (consider that I do not need the output from the server) in the setUp method, use it in my test suite and then terminate it in the tearDown method, like this:
class TestActuator(unittest.TestCase):
"""Class to test the actuator module"""
def setUp(self):
"""
Setup a real Actuator object with a modbus server running
in background for tests
"""
modbus_server_path = "path/to/modbus_server.py"
cmd = "python3 {}".format(modbus_server_path)
self.modbus_server = Popen(cmd.split(), shell=False, stdout=DEVNULL, stderr=DEVNULL)
# other stuff
def test1(self):
def test2(self):
def tearDown(self):
"""Cleanup everything before exiting"""
self.modbus_server.terminate()
if __name__ == '__main__':
unittest.main()
Unfortunately, it seems more challenging than I expected.
Everything I tried from other topics on stackoverflow failed:
Use Popen.kill() instead of terminate. I tried also to "del" after kill or terminate or to use os.kill(self.modbus_server.pid, SIGTERM) instead.
Add or change args to the Popen command, such as shell=True instead of shell=False and close_fds=True.
Use the other variants of subprocess Popen, like check_output, run, call, etc...
Use os.spawnl instead of subprocess Popen.
None of these worked. What happens most of the times is that the server fails to start properly, so all the other tests fail and the modbus_server process is not terminated (I have to kill it manually).
Do you have any idea? Thank you.
I will share how I solved my issue and what I learned so far:
About the attempt to make the modbus TCP server running as a subprocess while running tests, you need to change a few things to make it work decently.
The first thing is to use "setUpClass" and "tearDownClass" instead of setUp and tearDown, (you can use also both of them but in my case I needed the first two) so that you can run the setup of your server just once for all the test suite and not before and after each test case. Unfortunately, the syntax for setUpClass and tearDownClass is not so straightforward. This answer helped me a lot: Run setUp only once for a set of automated tests
The second thing is about the TCP socket. Basically, I wanted to be sure that my tests worked fine and run them a few times in a row. So, sometimes everything worked fine, other times they failed. After a while I found that if I run the tests without waiting at least one minute before trying again, they will fail.
This is due to the TIME_WAIT (set to 60 seconds by default) of the TCP socket bound by the modbus server on the local address and port 5020 in my case. If you want to re-bind a socket on the same TCP address and port you used before, you will have to wait 60 seconds. This answer was useful to understand why this happens: Setting TIME_WAIT TCP
After I figure out how to make all that stuff work together, I decided to use mocks because my solution seemed too hacky for my tastes.
I am currently working on a pre-flight-check script for the Apache PLC4X project. There I check the existence of required third party tools and their versions.
If I run "python --version" on the commandline, I get a nice response.
However if I run it in Groovy:
print "Detecting Python version: "
def output = ("python --version").execute().text
I just get an empty string.
All the other tools don't show this behavior. All others have the console output in "output".
How can I do the check I want to do? What am I doing wrong?
Don't assume everything you see on the terminal comes via standard output.
Informational messages are frequently sent to standard error instead, to avoid having them get caught in any processing pipelines (which was why the two channels were created way back in the early UNIX days).
I'm working on a server bot in python3 (using asyncio), and I would like to incorporate an update function for collaborators to instantly test their contributions. It is hosted on a VPS that I access via ssh. I run the process in tmux and it is often difficult for other contributors to relaunch the script once they have made a commit, etc. I'm very new to python, and I just use what I can find. So far I have used subprocess.Popen to run git pull, but I have no way for it to automatically restart the script.
Is there any way to terminate a running asyncio loop (ideally without errors) and restart it again?
You can not start a event loop stopped by event_loop.stop()
And in order to incorporate the changes you have to restart the script anyways (some methods might not exist on the objects you have, etc.)
I would recommend something like:
asyncio.ensure_future(git_tracker)
async def git_tracker():
# check for changes in version control, maybe wait for a sync point and then:
sys.exit(0)
This raises SystemExit, but despite that exits the program cleanly.
And around the python $file.py a while true; do git pull && python $file.py ; done
This is (as far as I know) the simplest approach to solve your problem.
For your use case, to stay on the safe side, you would probably need to kill the process and relaunch it.
See also: Restart process on file change in Linux
As a necromancer, I thought I give an up-to-date solution which we use in our UNIX system.
Using the os.execl function you can tell python to replace the current process with a new one:
These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller. Errors will be reported as OSError exceptions.
In our case, we have a bash script which executes the killall python3.7, sending the SIGTERM signal to our python apps which in turn listen to it via the signal module and gracefully shutdown:
loop = asyncio.get_event_loop()
loop.call_soon_threadsafe(loop.stop)
sys.exit(0)
The script than starts the apps in background and finishes.
Note that killall python3.7 will send SIGTERM signal to every python3.7 process!
When we need to restart we jus rune the following command:
os.execl("./restart.sh", 'restart.sh')
The first parameter is the path to the file and the second is the name of the process.
I'm writing a CGI script that is supposed to send data to a user until they disconnect, then run logging tasks afterwards.
THE PROBLEM: Instead of break executing and the logging getting completed when the client disconnects (detected by inability to write to the stdout buffer), the script ends or is killed (I cannot find any logs anywhere for how this exit is occurring)
Here is a snippet of the code:
for block in r.iter_content(262144):
if stopRecord == True:
r.close()
if not block:
break
if not sys.stdout.buffer.write(block): #The code fails here after a client disconnects
break
cacheTemp.close()
####write data to other logs and exit gracefully####
I have tried using "except:" as well as "except SystemExit:" but to no avail. Has anyone been able to solve this problem? (It is for a CGI script which is supposed to log when the client terminates their connection)
UPDATE: I have now tried using signal to interrupt the kill process in the script, which also didn't work. Where can I see an error log? I know exactly which line fails and under which conditions, but there is no error log or anything like I would get if I ran a script which failed in a terminal.
When you say it kills the program, you mean the main python process exits - and not by some thrown exception? That's kinda weird. A workaround might be to have the task run in a separate Thread or process, and then monitor that until it dies and subsequently execute the second task.
I have a node.js application, which connect everyday to a server.
On this server, a new version of the app can be available, if so, the installed app download it, check if the download is complete, and if so, stop itself calling a shell script, which replace the old app by the new one, and start it.
I m struggling at starting the update script.
I know I can start it with child_process_execFile function, which I do:
var execF = require('child_process').execFile;
var PATH = process.argv[1].substr(0, process.argv[1].lastIndexOf('/')+1),
filename = 'newapp.js',
execF(PATH + 'up.sh', [PATH + filename], function () {console.log('done'); return ;});
up.sh, for now is just:
cat $1 > /home/pi/test
I get 'done' printed in the console, but test isn t created.
I know that execFile create a subprocess, is it what block the script to do that?
If I suceed to start this, I know I only have to make some cp in the script to have my app auto-updating.
EDIT:
Started as usual (calling the script from console), it work well, is there a reason for the script to don t execute when called from node.js?
I'd suggest that you consider using a module that can do this for you automatically rather than duplicating the effort. Or, at least use their technique as inspiration for you own requirements.
One example is: https://github.com/edwardhotchkiss/always
It's simple to use:
Usage: always <app.js>
=> always app.js
Then, anytime your code changes, the app is killed, and restarted.
As you can see in the source, it uses the Monitor class to watch a specified file, and then uses spawn to kick it off (and of course kill to end the process when a change has happened).
Unfortunately, the [always] output is currently hardcoded into the code, but it would be a simple change/pull request I'm sure to make it optional/configurable. If the author doesn't accept your change, you could just modify a local copy of the code (as it's quite simple overall).
Make sure when you spawn/exec the process you are executing the shell that will be processing the script and not the script itself.
Should be something like
execF("/usr/bin/sh", [PATH + 'up.sh', PATH + filename]);