Is this a Python 3 regression in IPython Notebook? - python-3.x

I am attempting to create some simple asynchronously-executing animations based on ipythonblocks and I am trying to update the cell output area using clear_output() followed by a grid.show().
For text output the basis of the technique is discussed in Per-cell output for threaded IPython Notebooks so my simplistic assumption was to use the same method to isolate HTML output. Since I want to repeatedly replace a grid with its updated HTML version I try to use clear_output() to ensure that only one copy of the grid is displayed.
I verified that this proposed technique works for textual output with the following cells. First the context manager.
import sys
from contextlib import contextmanager
import threading
stdout_lock = threading.Lock()
n = 0
#contextmanager
def set_stdout_parent(parent):
"""a context manager for setting a particular parent for sys.stdout
(i.e. redirecting output to a specific cell). The parent determines
the destination cell of output
"""
global n
save_parent = sys.stdout.parent_header
# we need a lock, so that other threads don't snatch control
# while we have set a temporary parent
with stdout_lock:
sys.stdout.parent_header = parent
try:
yield
finally:
# the flush is important, because that's when the parent_header actually has its effect
n += 1; print("Flushing", n)
sys.stdout.flush()
sys.stdout.parent_header = save_parent
Then the test code
import threading
import time
class timedThread(threading.Thread):
def run(self):
# record the parent (uncluding the stdout cell) when the thread starts
thread_parent = sys.stdout.parent_header
for i in range(3):
time.sleep(2)
# then ensure that the parent is the same as when the thread started
# every time we print
with set_stdout_parent(thread_parent):
print(i)
timedThread().start()
This provided the output
0
Flushing 1
1
Flushing 2
2
Flushing 3
So I modified the code to clear the cell between cycles.
import IPython.core.display
class clearingTimedThread(threading.Thread):
def run(self):
# record the parent (uncluding the stdout cell) when the thread starts
thread_parent = sys.stdout.parent_header
for i in range(3):
time.sleep(2)
# then ensure that the parent is the same as when the thread started
# every time we print
with set_stdout_parent(thread_parent):
IPython.core.display.clear_output()
print(i)
clearingTimedThread().start()
As expected the output area of the cell was repeatedly cleared, and ended up reading
2
Flushing 6
I therefore thought I was on safe ground in using the same technique to clear a cell's output area when using ipythonblocks. Alas no. This code
from ipythonblocks import BlockGrid
w = 10
h = 10
class clearingBlockThread(threading.Thread):
def run(self):
grid = BlockGrid(w, h)
# record the parent (uncluding the stdout cell) when the thread starts
thread_parent = sys.stdout.parent_header
for i in range(10):
# then ensure that the parent is the same as when the thread started
# every time we print
with set_stdout_parent(thread_parent):
block = grid[i, i]
block.green = 255
IPython.core.display.clear_output(other=True)
grid.show()
time.sleep(0.2)
clearingBlockThread().start()
does indeed produce the desired end state (a black matrix with a green diagonal) but the intermediate steps don't appear in the cell's output area. To complicate things slightly (?) this example is running on Python 3. In checking before posting here I discover that the expected behavior (a simple animation) does in fact occur under Python 2.7. Hence I though to ask whether this is an issue I need to report.

Related

Continuous communication between parent and child subprocess in Python (Windows)?

I have this script:
import subprocess
p = subprocess.Popen(["myProgram.exe"],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
while True:
out, _ = p.communicate(input().encode())
print(out.decode())
which works fine until the second input where I get:
ValueError: Cannot send input after starting communication
Is there a way to have multiple messages sent between the parent and child process in Windows ?
[EDIT]
I don't have access to the source code of myProgram.exe
It is an interactive command line application returning results from queries
Running >> myProgram.exe < in.txt > out.txt works fine with in.txt:
query1;
query2;
query3;
Interacting with another running process via stdin/stdout
To simulate the use case where a Python script starts a command line interactive process and sends/receives text over stdin/stdout, we have a primary script that starts another Python process running a simple interactive loop.
This can also be applied to cases where a Python script needs to start another process and just read its output as it comes in without any interactivity beyond that.
primary script
import subprocess
import threading
import queue
import time
if __name__ == '__main__':
def enqueue_output(outp, q):
for line in iter(outp.readline, ''):
q.put(line)
outp.close()
q = queue.Queue()
p = subprocess.Popen(["/usr/bin/python", "/test/interact.py"],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
# stderr = subprocess.STDOUT,
bufsize = 1,
encoding ='utf-8')
th = threading.Thread(target=enqueue_output, args=(p.stdout, q))
th.daemon = True
th.start()
for i in range(4):
print("dir()", file=p.stdin)
print(f"Iteration ({i}) Parent received: {q.get()}", end='')
# p.stdin.write("dir()\n")
# while q.empty():
# time.sleep(0)
# print(f"Parent: {q.get_nowait()}")
interact.py script
if __name__ == '__main__':
for i in range(2):
cmd = raw_input()
print("Iteration (%i) cmd=%s" % (i, cmd))
result = eval(cmd)
print("Iteration (%i) result=%s" % (i, str(result)))
output
Iteration (0) Parent received: Iteration (0) cmd=dir()
Iteration (1) Parent received: Iteration (0) result=['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'cmd', 'i']
Iteration (2) Parent received: Iteration (1) cmd=dir()
Iteration (3) Parent received: Iteration (1) result=['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'cmd', 'i', 'result']
This Q&A was leveraged to simulate non-blocking reads from the target process: https://stackoverflow.com/a/4896288/7915759
This method provides a way to check for output without blocking in the main thread; q.empty() will tell you if there's no data. You can play around with blocking calls too using q.get() or with a timeout q.get(2) - the parameter is number of seconds. It can be a float value less than zero.
Text based interaction between processes can be done without the thread and queue, but this implementation gives more options on how to retrieve the data coming back.
The Popen() parameters, bufsize=1 and encoding='utf-8' make it possible to use <stdout>.readline() from the primary script and sets the encoding to an ascii compatible codec understood by both processes (1 is not the size of the buffer, it's a symbolic value indicating line buffering).
With this configuration, both processes can simply use print() to send text to each other. This configuration should be compatible for a lot of interactive text based command line tools.

Transfer of data between Python files

Need some help please to explain why the following does not work.
Environment: Python 3.4, Gtk3.0, limited experience of Python
File selectcontact.py contains code to select one of a number of records and pass its key back to its parent process for use in one of at least three other actions.
Code snippet from the parent class:
….
self.cindex = 0
….
def editcontact_clicked (self, menuitem):
import selectcontact
selectcontact.SelectContactGUI(self)
print ('From Manage ', self.cindex)
if self.cindex > 0:
import editcontact
editcontact.EditContactGUI(self.db, self.cindex)
….
Code snippet from selectcontact:
class SelectContactGUI:
def init(self, parent_class):
self.builder = Gtk.Builder()
self.builder.add_from_file(UI_FILE)
self.builder.connect_signals(self)
self.parent_class = parent_class
self.db = parent_class.db
self.cursor = self.db.cursor(cursor_factory = psycopg2.extras.NamedTupleCursor)
self.contact_store = self.builder.get_object('contact_store')
self.window = self.builder.get_object('window1')
self.window.show_all()
def select_contact_path(self, path):
self.builder.get_object('treeview_selection1').select_path(path)
def contact_treerow_changed (self, treeview):
selection = self.builder.get_object('treeview_selection1')
model, path = selection.get_selected()
if path != None:
self.parent_class.cindex = model[path][0]
print ('From select ', self.parent_class.cindex)
self.window.destroy()
….
window1 is declared as “modal”, so I was expecting the call to selectcontact to act as a subroutine, so that editcontact wouldn’t be called until control was passed back to the parent. The parent_class bit works because the contact_store is correctly populated. However the transfer back to the parent appears not to work, and the two print statements occur in the wrong order:
From Manage 0
From select 2
Comments gratefully received.
Graeme
"Modal" refers to windows only. That is, a modal window prevents accessing the parent window.
It has little to do with what code is running. I am not familiar with this particular windowing framework, but any I have worked with has had a separate thread for GUI and at least one for processing, to keep the GUI responsive, and message loops running in all active windows, not just the one currently with the focus. The modal dialog has no control over what code in other threads are executed when.
You should be able to break into the debugger and see what threads are running and what is running in each thread at any given time.

Multiprocessing - tkinter pipeline communication

I have a question on multiprocessing and tkinter. I am having some problems getting my process to function parallel with the tkinter GUI. I have created a simple example to practice and have been reading up to understand the basics of multiprocessing. However when applying them to tkinter, only one process runs at the time. (Using Multiprocessing module for updating Tkinter GUI) Additionally, when I added the queue to communicate between processes, (How to use multiprocessing queue in Python?), the process won't even start.
Goal:
I would like to have one process that counts down and puts the values in the queue and one to update tkinter after 1 second and show me the values.
All advice is kindly appreciated
Kind regards,
S
EDIT: I want the data to be available when the after method is being called. So the problem is not with the after function, but with the method being called by the after function. It will take 0.5 second to complete the calculation each time. Consequently the GUI is unresponsive for half a second, each second.
EDIT2: Corrections were made to the code based on the feedback but this code is not running yet.
class Countdown():
"""Countdown prior to changing the settings of the flows"""
def __init__(self,q):
self.master = Tk()
self.label = Label(self.master, text="", width=10)
self.label.pack()
self.counting(q)
# Countdown()
def counting(self, q):
try:
self.i = q.get()
except:
self.label.after(1000, self.counting, q)
if int(self.i) <= 0:
print("Go")
self.master.destroy()
else:
self.label.configure(text="%d" % self.i)
print(i)
self.label.after(1000, self.counting, q)
def printX(q):
for i in range(10):
print("test")
q.put(9-i)
time.sleep(1)
return
if __name__ == '__main__':
q = multiprocessing.Queue()
n = multiprocessing.Process(name='Process2', target=printX, args = (q,))
n.start()
GUI = Countdown(q)
GUI.master.mainloop()
Multiprocessing does not function inside of the interactive Ipython notebook.
Multiprocessing working in Python but not in iPython As an alternative you can use spyder.
No code will run after you call mainloop until the window has been destroyed. You need to start your other process before you call mainloop.
You are calling wrong the after function. The second argument must be the name of the function to call, not a call to the function.
If you call it like
self.label.after(1000, self.counting(q))
It will call counting(q) and wait for a return value to assign as a function to call.
To assign a function with arguments the syntax is
self.label.after(1000, self.counting, q)
Also, start your second process before you create the window and call counting.
n = multiprocessing.Process(name='Process2', target=printX, args = (q,))
n.start()
GUI = Countdown(q)
GUI.master.mainloop()
Also you only need to call mainloop once. Either position you have works, but you just need one
Edit: Also you need to put (9-i) in the queue to make it count down.
q.put(9-i)
Inside the printX function

Keep GIF animation running while doing calculations

I am trying to improve the user experience by showing a load mask above the active QMainWindow/QDialog when performing tasks that takes some time. I have managed to get it working as I want it, except for a moving GIF when performing the task. If I leave the load mask on after the task is complete, the GIF starts moving as it should.
My class for the load mask:
from PyQt4 import QtGui, QtCore
from dlgLoading_view import Ui_dlgLoading
class dlgLoading(QtGui.QDialog, Ui_dlgLoading):
def __init__(self,parent):
QtGui.QDialog.__init__(self,parent)
self.setupUi(self)
self.setWindowFlags(QtCore.Qt.WindowFlags(QtCore.Qt.FramelessWindowHint))
self.setGeometry(0, 0, parent.frameGeometry().width(), parent.frameGeometry().height())
self.setStyleSheet("background-color: rgba(255, 255, 255, 100);")
movie = QtGui.QMovie("loader.gif")
self.lblLoader.setMovie(movie)
movie.start()
def showEvent(self, event):
QtGui.qApp.processEvents()
super(dlgLoading, self).showEvent(event)
def setMessage(self,message):
self.lblMessage.setText(message)
The Ui_dlgLoading contains two labels and some vertical spacers: lblLoader (will contain the gif) and lblMessage (will contain a message if needed)
I create the load mask with this code:
loadmask = dlgLoading(self)
loadmask.setMessage('Reading data... Please wait')
loadmask.show()
I figured I needed some multithreading/multiprocessing, but I can't for the life of me figure out how to do it. I read somewhere that you can't tamper with the GUIs threading, so I would need to move the heavy task there instead, but I'm still blank.
As a simple example, let's say I am trying to load a huge file into memory:
file = open(dataFilename, 'r')
self.dataRaw = file.read()
file.close()
Around that I would create and close my load mask dialog. How do I start the file read without halting the GIF animation?
The GUI is for running some heavy external exe files, so it should work with that too.
I ended up doing this:
class runthread(threading.Thread):
def __init__(self, commandline, cwd):
self.stdout = None
self.stderr = None
self.commandline = commandline
self.cwd = cwd
self.finished = False
threading.Thread.__init__(self)
def run(self):
subprocess.call(self.commandline, cwd=self.cwd)
self.finished = True
class command()
def __init__(self):
...
def run():
...
thread = runthread("\"%s\" \"%s\"" % (os.path.join(self.__caller.exefolder, "%s.exe" % self.__cmdtype), self.__name), self.__caller.exeWorkdir)
thread.start()
count = 0
sleeptime = 0.5
maxcount = 60.0/sleeptime
while True:
time.sleep(sleeptime)
QtWidgets.qApp.processEvents()
count += 1
if thread.finished:
break
if count >= maxcount:
results = QtWidgets.QMessageBox.question(self.__caller, "Continue?", "The process is taking longer than expected. Do you want to continue?", QtWidgets.QMessageBox.Yes | QtWidgets.QMessageBox.No)
if results == QtWidgets.QMessageBox.Yes:
count == 0
else:
QtWidgets.QMessageBox.warning(self.__caller, "Process stopped", "The process was stopped")
return False
It actually doesn't directly answer my question, but it worked for me, so I'm posting the answer if others want to do something similar.
I call a process (in this case Pythons subprocess.call) through a thread and track when the process is actually finished. A continuous loop checks periodically if the process is done and updates the GUI (processEvents - this is what triggers the GIF to update). To avoid an infinite loop I offer the user an option to exit after some time.

PyQtGraph: GUI unresponsive during only one process

I'm writing a GUI for a video camera that can basically run in two modes that I call liveview and recordview. The only difference being that I'm recording in the latter and only viewing in the former.
In liveview mode the image gets updated properly. I've set a button that triggers recordview but during this acquisition the GUI gets unresponsive and the image doesn't get updated. Let me show you the relevant parts of the code:
import numpy as np
from PyQt4 import QtGui, QtCore
import pyqtgraph as pg
from lantz.drivers.andor.ccd import CCD
app = QtGui.QApplication([])
def updateview(): # <-- this works OK
global img, andor
img.setImage(andor.most_recent_image16(andor.detector_shape),
autoLevels=False)
def liveview():
""" Image live view when not recording
"""
global andor, img, viewtimer
andor.acquisition_mode = 'Run till abort'
andor.start_acquisition()
viewtimer.start(0)
def UpdateWhileRec():
global stack, andor, img, n, ishape
j = 0
while j < n:
if andor.n_images_acquired > j:
# Data saving <-- this part (and the whole while-loop) works OK
i, j = andor.new_images_index
stack[i - 1:j] = andor.images16(i, j, ishape, 1, n)
# Image updating <-- this doesn't work
img.setImage(stack[j - 1], autoLevels=False)
liveview() # After recording, it goes back to liveview mode
def record(n):
""" Record an n-frames acquisition
"""
global andor, ishape, viewtimer, img, stack, rectimer
andor.acquisition_mode = 'Kinetics'
andor.set_n_kinetics(n)
andor.start_acquisition()
# Stop the QTimer that updates the image with incoming data from the
# 'Run till abort' acquisition mode.
viewtimer.stop()
QtCore.QTimer.singleShot(1, UpdateWhileRec)
if __name__ == '__main__':
with CCD() as andor:
win = QtGui.QWidget()
rec = QtGui.QPushButton('REC')
imagewidget = pg.GraphicsLayoutWidget()
p1 = imagewidget.addPlot()
img = pg.ImageItem()
p1.addItem(img)
layout = QtGui.QGridLayout()
win.setLayout(layout)
layout.addWidget(rec, 2, 0)
layout.addWidget(imagewidget, 1, 2, 3, 1)
win.show()
viewtimer = QtCore.QTimer()
viewtimer.timeout.connect(updateview)
# Record routine
n = 100
newimage = np.zeros(ishape)
stack = np.zeros((n, ishape[0], ishape[1]))
rec.pressed.connect(lambda: record(n))
liveview()
app.exec_()
viewtimer.stop()
As you see UpdateWhileRec runs only once per acquisition while updateview runs until viewtimer.stop() is called.
I'm new to PyQt and PyQtGraph so regardless of the particular way of solving my present issue, there's probably a better way to do everything else. If that's the case please tell me!
thanks in advanced
Your problem stems from the fact that you need to return control to the Qt event loop for it to redraw the picture. Since you remain in the UpdateWhileRec callback while waiting for the next image to be acquired, Qt never gets a chance to draw the image. It only gets the chance once you exit the function UpdateWhileRec.
I would suggest the following changes.
Then instead of your while loop in UpdateWhileRec, have a QTimer that periodically calls the contents of your current while loop (i would probably suggest a singleshot timer). This ensures control will be returned to Qt so it can draw the image before checking for a new one.
So something like:
def UpdateWhileRec():
# Note, j should be initialised to 0 in the record function now
global stack, andor, img, n, j, ishape
if andor.n_images_acquired > j:
# Data saving <-- this part (and the whole while-loop) works OK
i, j = andor.new_images_index
stack[i - 1:j] = andor.images16(i, j, ishape, 1, n)
# Image updating <-- this should now work
img.setImage(stack[j - 1], autoLevels=False)
if j < n:
QTimer.singleShot(0, UpdateWhileRec)
else:
liveview() # After recording, it goes back to liveview mode
Note, you should probably put functions and variables in a class, and create an instance of the class (an object). That way you don't have to call global everywhere and things are more encapsulated.
Ultimately, you may want to look into whether your andor library supports registering a function to be called when a new image is available (a callback) which would save you doing this constant polling and/or acquiring the images in a thread and posting them back to the GUI thread to be drawn. But one step at a time!

Resources