The reason I am trying to do this is because Windows is terribly slow at opening and closing processes all the time. Its terribly inefficient which is the anti-purpose of multiprocessing. If one could start say 10 processes that do a specific operation on some data that can return the data (with queue or something) without closing it so you can send it more data to operate on and return. Basically a hive of processes holding functions that are always open ready to process and return data without death on the return so you don't have to keep opening new ones. I guess I could just run Linux Debian Crunchbang but I want to make it Windows efficient as well.
import multiprocessing
def operator_dies(data, q):
for i in range(len(data)):
if data % i == 0:
return q.put(0) # Function and process die here
return q.put(1) # And here
def operator_lives(data, q):
for i in range(len(data)):
if data % i == 0:
q.put(0) # I want it to stay open, but send data back with q.put()
q.put(1) # Same
def initialize(data):
q = multiprocessing.Queue()
p = [multiprocessing.Process(target=operator_dies, args=data, q]
p.daemon = True
p.start()
# Multiprocessing.Process will open more processes.
# Is there another multiprocessing function that can send new data as
# arguments in the function of an already open process ID and reset its loop?
# I cannot simply do: q.put().processid from here to refresh
# a variable for arguments in an open process can I?
if __name__ == '__main__':
initialize(data)
Ok so I'm trying to run a C program from a python script. Currently I'm using a test C program:
#include <stdio.h>
int main() {
while (1) {
printf("2000\n");
sleep(1);
}
return 0;
}
To simulate the program that I will be using, which takes readings from a sensor constantly.
Then I'm trying to read the output (in this case "2000") from the C program with subprocess in python:
#!usr/bin/python
import subprocess
process = subprocess.Popen("./main", stdout=subprocess.PIPE)
while True:
for line in iter(process.stdout.readline, ''):
print line,
but this is not working. From using print statements, it runs the .Popen line then waits at for line in iter(process.stdout.readline, ''):, until I press Ctrl-C.
Why is this? This is exactly what most examples that I've seen have as their code, and yet it does not read the file.
Is there a way of making it run only when there is something to be read?
It is a block buffering issue.
What follows is an extended for your case version of my answer to Python: read streaming input from subprocess.communicate() question.
Fix stdout buffer in C program directly
stdio-based programs as a rule are line buffered if they are running interactively in a terminal and block buffered when their stdout is redirected to a pipe. In the latter case, you won't see new lines until the buffer overflows or flushed.
To avoid calling fflush() after each printf() call, you could force line buffered output by calling in a C program at the very beginning:
setvbuf(stdout, (char *) NULL, _IOLBF, 0); /* make line buffered stdout */
As soon as a newline is printed the buffer is flushed in this case.
Or fix it without modifying the source of C program
There is stdbuf utility that allows you to change buffering type without modifying the source code e.g.:
from subprocess import Popen, PIPE
process = Popen(["stdbuf", "-oL", "./main"], stdout=PIPE, bufsize=1)
for line in iter(process.stdout.readline, b''):
print line,
process.communicate() # close process' stream, wait for it to exit
There are also other utilities available, see Turn off buffering in pipe.
Or use pseudo-TTY
To trick the subprocess into thinking that it is running interactively, you could use pexpect module or its analogs, for code examples that use pexpect and pty modules, see Python subprocess readlines() hangs. Here's a variation on the pty example provided there (it should work on Linux):
#!/usr/bin/env python
import os
import pty
import sys
from select import select
from subprocess import Popen, STDOUT
master_fd, slave_fd = pty.openpty() # provide tty to enable line buffering
process = Popen("./main", stdin=slave_fd, stdout=slave_fd, stderr=STDOUT,
bufsize=0, close_fds=True)
timeout = .1 # ugly but otherwise `select` blocks on process' exit
# code is similar to _copy() from pty.py
with os.fdopen(master_fd, 'r+b', 0) as master:
input_fds = [master, sys.stdin]
while True:
fds = select(input_fds, [], [], timeout)[0]
if master in fds: # subprocess' output is ready
data = os.read(master_fd, 512) # <-- doesn't block, may return less
if not data: # EOF
input_fds.remove(master)
else:
os.write(sys.stdout.fileno(), data) # copy to our stdout
if sys.stdin in fds: # got user input
data = os.read(sys.stdin.fileno(), 512)
if not data:
input_fds.remove(sys.stdin)
else:
master.write(data) # copy it to subprocess' stdin
if not fds: # timeout in select()
if process.poll() is not None: # subprocess ended
# and no output is buffered <-- timeout + dead subprocess
assert not select([master], [], [], 0)[0] # race is possible
os.close(slave_fd) # subproces don't need it anymore
break
rc = process.wait()
print("subprocess exited with status %d" % rc)
Or use pty via pexpect
pexpect wraps pty handling into higher level interface:
#!/usr/bin/env python
import pexpect
child = pexpect.spawn("/.main")
for line in child:
print line,
child.close()
Q: Why not just use a pipe (popen())? explains why pseudo-TTY is useful.
Your program isn't hung, it just runs very slowly. Your program is using buffered output; the "2000\n" data is not being written to stdout immediately, but will eventually make it. In your case, it might take BUFSIZ/strlen("2000\n") seconds (probably 1638 seconds) to complete.
After this line:
printf("2000\n");
add
fflush(stdout);
See readline docs.
Your code:
process.stdout.readline
Is waiting for EOF or a newline.
I cannot tell what you are ultimately trying to do, but adding a newline to your printf, e.g., printf("2000\n");, should at least get you started.
I'm attempting to have pexpect begin running a command which basically continually outputs some information every few milliseconds until cancelled with Ctrl + C.
I've attempted getting pexpect to log to a file, though these outputs are simply ignored and are never logged.
child = pexpect.spawn(command)
child.logfile = open('mylogfile.txt', 'w')
This results in the command being logged with an empty output.
I have also attempted letting the process run for a few seconds, then sending an interrupt to see if that logs the data, but this again, results in an almost empty log.
child = pexpect.spawn(command)
child.logfile = open('mylogfile.txt', 'w')
time.sleep(5)
child.send('\003')
child.expect('$')
This is the data in question:
image showing the data constantly printing to the terminal
I've attempted the solution described here: Parsing pexpect output though it hasn't worked for me and results in a timeout.
Managed to get it working by using Python Subprocess for this, not sure of a way to do it with Pexpect, but this got what I described.
def echo(self, n_lines):
output = []
if self.running is False:
# start shell
self.current_shell = Popen(cmd, stdout=PIPE, shell=True)
self.running = True
i = 0
# Read the lines from stdout, break after reading the desired amount of lines.
for line in iter(self.current_shell.stdout.readline, ''):
output.append(line[0:].decode('utf-8').strip())
if i == n_lines:
break
i += 1
return output
(in Python 3.5)
I am having difficulties to print stdout line by line (while running the program), and maintain the timeout function (to stop the program after sometime).
I have:
import subprocess as sub
import io
file_name = 'program.exe'
dir_path = r'C:/directory/'
p = sub.Popen(file_name, cwd = dir_path, shell=True, stdout = sub.PIPE, stderr = sub.STDOUT)
And while running "p", do these 2 things:
for line in io.TextIOWrapper(p.stdout, encoding="utf-8"):
print(line)
And do:
try:
outs = p.communicate(timeout=15) # Just to use timeout
except Exception as e:
print(str(e))
p.kill()
The program should print every output line but should not run the simulation for more than 15 seconds.
If I use the "p.communicate" before the "p.stdout", it will wait for the timeout ou the program to finish. If I use it on the other way, the program will not count the timeout.
I would like to do it without threading, and if possible without io too, it seems to be possible, but I don´t know how (need more practice and study). :-(
PS: The program I am running was written in fortran and is used to simulate water flow. If I run the exe from windows, it opens a cmd and prints a line on each timestep. And I am doing a sensitivity analysis changing the inputs on exe file.
That's because your process\child processes are not getting killed correctly
just modify your try,except as below
try:
pid_id=p.pid
outs = p.communicate(timeout=15) # Just to use timeout
except Exception as e:
print(str(e))
import subprocesss
#This will kill all the process and child process associated with p forcefully
subprocess.Popen('taskkill /F /T /PID %i' % pid_id)
I am attempting to create some simple asynchronously-executing animations based on ipythonblocks and I am trying to update the cell output area using clear_output() followed by a grid.show().
For text output the basis of the technique is discussed in Per-cell output for threaded IPython Notebooks so my simplistic assumption was to use the same method to isolate HTML output. Since I want to repeatedly replace a grid with its updated HTML version I try to use clear_output() to ensure that only one copy of the grid is displayed.
I verified that this proposed technique works for textual output with the following cells. First the context manager.
import sys
from contextlib import contextmanager
import threading
stdout_lock = threading.Lock()
n = 0
#contextmanager
def set_stdout_parent(parent):
"""a context manager for setting a particular parent for sys.stdout
(i.e. redirecting output to a specific cell). The parent determines
the destination cell of output
"""
global n
save_parent = sys.stdout.parent_header
# we need a lock, so that other threads don't snatch control
# while we have set a temporary parent
with stdout_lock:
sys.stdout.parent_header = parent
try:
yield
finally:
# the flush is important, because that's when the parent_header actually has its effect
n += 1; print("Flushing", n)
sys.stdout.flush()
sys.stdout.parent_header = save_parent
Then the test code
import threading
import time
class timedThread(threading.Thread):
def run(self):
# record the parent (uncluding the stdout cell) when the thread starts
thread_parent = sys.stdout.parent_header
for i in range(3):
time.sleep(2)
# then ensure that the parent is the same as when the thread started
# every time we print
with set_stdout_parent(thread_parent):
print(i)
timedThread().start()
This provided the output
0
Flushing 1
1
Flushing 2
2
Flushing 3
So I modified the code to clear the cell between cycles.
import IPython.core.display
class clearingTimedThread(threading.Thread):
def run(self):
# record the parent (uncluding the stdout cell) when the thread starts
thread_parent = sys.stdout.parent_header
for i in range(3):
time.sleep(2)
# then ensure that the parent is the same as when the thread started
# every time we print
with set_stdout_parent(thread_parent):
IPython.core.display.clear_output()
print(i)
clearingTimedThread().start()
As expected the output area of the cell was repeatedly cleared, and ended up reading
2
Flushing 6
I therefore thought I was on safe ground in using the same technique to clear a cell's output area when using ipythonblocks. Alas no. This code
from ipythonblocks import BlockGrid
w = 10
h = 10
class clearingBlockThread(threading.Thread):
def run(self):
grid = BlockGrid(w, h)
# record the parent (uncluding the stdout cell) when the thread starts
thread_parent = sys.stdout.parent_header
for i in range(10):
# then ensure that the parent is the same as when the thread started
# every time we print
with set_stdout_parent(thread_parent):
block = grid[i, i]
block.green = 255
IPython.core.display.clear_output(other=True)
grid.show()
time.sleep(0.2)
clearingBlockThread().start()
does indeed produce the desired end state (a black matrix with a green diagonal) but the intermediate steps don't appear in the cell's output area. To complicate things slightly (?) this example is running on Python 3. In checking before posting here I discover that the expected behavior (a simple animation) does in fact occur under Python 2.7. Hence I though to ask whether this is an issue I need to report.