Compiled Script Using Multiprocessing Crashes When Run - python-3.x

I wrote a simple script using Multiprocessing to fetch corresponding records from multiple databases, each containing different info for the same record. The script runs fine on my IDE and cmd, but whenever I tried to compile it with PyInstaller I got instant black screen followed by the sound of tons of error messages.
This is based on Python36, involving psycopg2, fiona and shapely as the files generated are shapefiles. But I compiled other scripts involving these modules before and they work fine.
import psycopg2
import fiona
from shapely.geometry import mapping
from shapely.wkt import loads
import re
import numpy
from multiprocessing import Process
def A(list):
for file in list:
query database A, generate shapefile for product A
def B(list):
for file in list:
query database B, generate shapefile for product B
def C(list):
for file in list:
query database C, generate shapefile for product C
if __name__ == '__main__':
e = Process(target=A, args=(scopes,))
m = Process(target=B, args=(scopes,))
p = Process(target=C, args=(scopes,))
e.start()
m.start()
p.start()
e.join()
print("\nProduct A is done")
m.join()
print("Product B is done")
p.join()
print("Product C is done")
The spec generated during the compiling are like this:
# -*- mode: python -*-
block_cipher = None
a = Analysis(['Scope Process Monitor.py'],
pathex=['C:\\Python36-32\\Lib\\site-packages\\osgeo', 'C:\\Users\\myname\\OneDrive - Company\\Scripts\\Scope Process Monitor'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='Scope Process Monitor',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True )
I am guessing I need to limit the use of CPU in the script somewhere, but this is actually my first time trying Multiprocessing module, so I do not know how.

Related

OSError: Can't get source for <function swish_jit at 0x000001BA57F0C288>. TorchScript requires source access

I am trying to create an executable. My code test.py works well when running with python, but the generated executable file fails with OsError error. I tried to add some hiddenimports and data to the spec file, but I am still getting the same error.
You can find my test.py and test.spec files here.
test.py
from lungmask import mask
import SimpleITK as sitk
import nibabel as nib
import glob
import shutil
from ttictoc import tic,toc
import os
import torch.multiprocessing as mp
import sys
import cv2
import numpy as np
import albumentations as albu
import torch
import segmentation_models_pytorch as smp
import efficientnet_pytorch
if __name__ == '__main__':
print('hello world')
test.spec
# -*- mode: python ; coding: utf-8 -*-
from pathlib import Path
import torch
import inspect
import efficientnet_pytorch
from os import path
block_cipher = None
#Excluded_modules = ['Torch.Distributions'] # Add this line
site_packages='C:\\Users\\pc\\Anaconda3\\envs\\pytorch-3.7\\Lib\\site-packages'
def collect_source_files(modules):
datas = []
for module in modules:
source = inspect.getsourcefile(module)
dest = f"src.{module.__name__}" # use "src." prefix
datas.append((source, dest))
return datas
a = Analysis(['test.py'],
pathex=['E:\\COVID-19\\New folder'],
binaries=[('C:\\Users\\pc\\Anaconda3\\envs\\pytorch-3.7\\Lib\\site-packages\\torch\\lib\\caffe2_nvrtc.dll','.')],
datas=[(path.join(site_packages,"torch"), "torch"),
(path.join(site_packages,"torchvision"), "torchvision"),
(path.join(site_packages,"efficientnet_pytorch"), "efficientnet_pytorch")],
hiddenimports=['pydicom', 'pydicom.encoders', 'pydicom.encoders.gdcm', 'pydicom.encoders.pylibjpeg', 'fill_voids', 'fastremap', 'nibabel', 'nibabel.nifti1','torch','av', 'fvcore','torchvision', 'detectron2','efficientnet_pytorch','segmentation_models_pytorch'],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes= [], #Excluded_modules
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
MISSING_LIBS = (
Path(r'C:\Windows\System32\nvcuda.dll'),
)
a.binaries += TOC([(lib.name, str(lib.resolve()),'BINARY') for lib in MISSING_LIBS])
MISSING_LIBS = (
Path(r'C:\Users\pc\Anaconda3\envs\pytorch-3.7\Library\bin\nvrtc64_101_0.dll'),
)
a.binaries += TOC([(lib.name, str(lib.resolve()),'BINARY') for lib in MISSING_LIBS])
source_files = collect_source_files([torch]) # return same structure as `collect_data_files()`
source_files_toc = TOC((name, path, 'DATA') for path, name in source_files)
pyz = PYZ(a.pure, a.zipped_data, source_files_toc,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='test',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True,
disable_windowed_traceback=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='test')

Import internal exe modules from an external subprocess module (pyinstaller)

I'd like to import internal exe modules from an external process. I can do this fine from an IDE, but when I package the project with pyinstaller, then run it, the modules can't be found.
The high level flow looks like this:
Importing maya_app from within the external userSetup.py returns an error: ImportError: No modules named maya_app
This post seems like it should help me but there's something about my setup that's preventing this solution from working. That or I'm just doing it wrong.
Is it even possible to import modules from one exe into another exe? Thanks!
Code:
launch_maya.py
import os
import sys
import subprocess
from PySide2 import QtWidgets
class Widget(QtWidgets.QWidget):
def __init__(self, parent=None):
super(Widget, self).__init__(parent)
# GUI
btn_launch = QtWidgets.QPushButton('launch maya')
btn_launch.clicked.connect(self.on_launch)
# Layout
main_layout = QtWidgets.QHBoxLayout(self)
main_layout.addWidget(btn_launch)
self.setLayout(main_layout)
# Root path exe vs ide
if getattr(sys, 'frozen', False):
self.root_path = sys._MEIPASS
else:
self.root_path = os.path.join(os.path.dirname(os.path.realpath(__file__)))
def _set_app_envs(self):
_envs = os.environ.copy()
_envs['MAYA_SCRIPT_PATH'] = os.path.join(self.root_path, 'scripts').replace('\\', '/')
# Python path envs
_python_path_list = [
'C:\\Program Files\\Autodesk\\Maya2020\\Python\\Lib\\site-packages',
'C:\\Program Files\\Autodesk\\Maya2020\\Python\\DLLs',
os.path.join(self.root_path, 'scripts').replace('\\', '/'),
os.path.join(self.root_path, 'internal_source', 'maya_app')
]
# PYTHONPATH exe vs ide
if getattr(sys, 'frozen', False):
_envs['PYTHONPATH'] = os.pathsep.join(_python_path_list)
_envs['PYTHONHOME'] = 'C:\\Program Files\\Autodesk\\Maya2020\\bin'
else:
_envs['PYTHONPATH'] += os.pathsep + os.pathsep.join(_python_path_list)
return _envs
def on_launch(self):
# Maya file path
file_path_abs = '{}/scenes/test.mb'.format(self.root_path).replace('\\', '/')
print(file_path_abs)
app_exe = r'C:/Program Files/Autodesk/Maya2020/bin/maya.exe'
_envs = self._set_app_envs()
if os.path.exists(file_path_abs):
proc = subprocess.Popen(
[app_exe, file_path_abs],
env=_envs,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
)
if __name__ == "__main__":
app = QtWidgets.QApplication(sys.argv)
window = Widget()
window.resize(400, 400)
window.show()
sys.exit(app.exec_())
bundle.spec
# -*- mode: python ; coding: utf-8 -*-
block_cipher = None
added_files = [
('./scenes', 'scenes'),
('./scripts', 'scripts')
]
a = Analysis(['launch_maya.py'],
pathex=[
'D:/GitStuff/mb-armada/example_files/exe_bundle',
'D:/GitStuff/mb-armada/dependencies/Qt.py',
'D:/GitStuff/mb-armada/venv/Lib/site-packages',
],
binaries=[],
datas=added_files,
hiddenimports=['internal_source', 'internal_source.maya_app'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='bundle',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='bundle')
maya_app.py
import os
import sys
import subprocess
from PySide2 import QtWidgets
class MainWidget(QtWidgets.QWidget):
def __init__(self, parent=None):
super(MainWidget, self).__init__(parent)
# GUI
btn_launch = QtWidgets.QPushButton('say hey')
btn_launch.clicked.connect(self.on_say_hey)
# Layout
main_layout = QtWidgets.QHBoxLayout(self)
main_layout.addWidget(btn_launch)
self.setLayout(main_layout)
print('I should be alive')
def on_say_hey(self):
print('hey')
if __name__ == "__main__":
app = QtWidgets.QApplication(sys.argv)
window = MainWidget()
window.resize(100, 100)
window.show()
sys.exit(app.exec_())
userSetup.py
import os
import sys
import maya.cmds as mc
print('hey')
def tweak_launch(*args):
print('Startup sequence running...')
os.environ['mickey'] = '--------ebae--------'
print(os.environ['mickey'])
root_path = os.getenv('_MMM_ROOT_PATH')
main_app_path = os.path.join(root_path, 'internal_source')
if not root_path in sys.path:
sys.path.append(main_app_path)
from internal_source import maya_app
w = maya_app.MainWidget()
w.show()
print('window should be up')
mc.evalDeferred("tweak_launch()")
At the highest level, this seems like it's really just about path management within your launched maya -- presumably the launcher app knows it's own paths, and the maya you want to fire off needs to be told about their existence.
The most reliable and least magic-prone way to do this is just to have your launcher unbundle any scripts scripts it needs to share with maya proper to a known location -- a hidden directory or even a session-specific temp directory will give you the maximum likelihood that you've got the latest version as your launcher revs.
When launching maya you can pass the paths a few ways -- probably the easiest is just to use site.addsitedir to put them on the path for that session rather than relying on Maya's many other possible search locations. You can launch the maya with the -c mel flag and pass a python command at startup, so you can control this behavior entirely from the launcher and not have to worry about interactions with userSetup.py. Something along these lines would make your stuff available to maya:
# unpack the 'payload' of scripts to share to a
# known location with something like pkgutil.get_data()
# https://docs.python.org/3/library/pkgutil.html
# note the single quotes -- you need them to handle escaping!
py_cmd = "import site; site.addsitedir('{}'); import my_startup_module"
python_to_execute = py_cmd.format(path_to_unpacked_modules)
# again, note annoying escapes
start_cmd = '\"python(\"{}\");\"'.format(python_to_execute)
maya_session = subprocess.Popen(['maya.exe', '-c', start_cmd])
If your exe has unpacked the files to path_to_unpacked_modules, maya will run that mel command, which will call python, add your module dir as a site-packages directory, and then import and run my_startup_module (from that location). This lets your launcher control the startup process and keeps you from having to fiddle separately with userSetup.py -- plus, since it does not touch env vars, you don't have to work to hard to run different sessions side by side.
If the code you want to share is just a bunch of modules (no binary extensions) you can keep your modules in a zip file and add that to the path -- python will be able to find modules inside the zip file automatically. That saves unpack time and also ensure you don't have leftover PYC files from older runs.
There's a lot of useful info in these threads:
https://tech-artists.org/t/python-maya-startup-script/2145/17
https://tech-artists.org/t/deploy-tools-for-your-maya-team/5029/13

Python stuck at last program execution

I am new to Python and I think I broke my python :(
I was trying Sentdex's PyQt4 YouTube tutorial right here.
I made the changes from PyQt4 to PyQt5. This is the code I was playing around. So I think, I messed up by printing the whole page on the console.
Now the output is:
Load finished
Look at you shinin!
Press any key to continue . . .
This is being shown for any code executed. That is python shows this code even if I try print("hello") in Visual code. I even tried to restart. Now like a virus, it is not clearing.
import bs4 as bs
import sys
import urllib.request
from PyQt5.QtWebEngineWidgets import QWebEnginePage
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl
class Page(QWebEnginePage):
def __init__(self, url):
self.app = QApplication(sys.argv)
QWebEnginePage.__init__(self)
self.html = ''
self.loadFinished.connect(self._on_load_finished)
self.load(QUrl(url))
self.app.exec_()
def _on_load_finished(self):
self.html = self.toHtml(self.Callable)
print('Load finished')
def Callable(self, html_str):
self.html = html_str
self.app.quit()
def main():
page = Page('https://pythonprogramming.net/parsememcparseface/')
soup = bs.BeautifulSoup(page.html, 'html.parser')
js_test = soup.find('p', class_='jstest')
print js_test.text
print (soup)
#js_test = soup.find('div', class_='aqi-meter-panel')
#display.popen.terminate()
if __name__ == '__main__': main()
OK, so finally got the problem fixed..went manually inside the temp files in C:\Users\xxx\AppData\Local and started on a deletion rampage...removed many files and folder remotely related to python,vscode and conda...this gave an error warning first time I executed my program again...then on subsequent run...no issue...python back to its normal self...surprised that I was not able to find any solution on the net for this.

Python suprocess not working after cx_freeze

this is my first question on stack-overflow!
I am currently writing a Python script (well, actually a few scripts) to manage book collections, that I would now like to freeze and distribute (this is my first 'big' project).
After looking at many options I decided to try with Cx_Freeze.
(I'm using Python 3.6 and Cx_Freeze 5.1.1).
In this project I often use 'subprocess' to move from a script to another.
In the interpreter it works just fine, if I let Cx_Freeze make the build folder using
python setup.py build
it works as well, but when I try to create a distributable file with
python setup.py bdist_msi
after installation it starts and works up to the first call for a subprocess, then nothing more.
Here is setup.py
from cx_Freeze import setup, Executable
import os.path
PYTHON_INSTALL_DIR = os.path.dirname(os.path.dirname(os.__file__))
os.environ['TCL_LIBRARY'] = os.path.join(PYTHON_INSTALL_DIR, 'tcl', 'tcl8.6')
os.environ['TK_LIBRARY'] = os.path.join(PYTHON_INSTALL_DIR, 'tcl', 'tk8.6')
setup(
name = "Libro",
version = "1.0.0",
options = {"build_exe": {
'packages': ["tkinter", "subprocess", ],
'include_files': [os.path.join(PYTHON_INSTALL_DIR, 'DLLs','tk86t.dll'), \
os.path.join(PYTHON_INSTALL_DIR, 'DLLs', 'tcl86t.dll'), \
'logo50x50.gif', 'check_1.txt', 'check_2.txt', 'start.py', \
'createdb.py', *and_a_few_more_files*],
'include_msvcr': True,
}},
executables = [Executable("Libro.py",base="Win32GUI")]
And this is the Libro.py script that becomes the executable.
#This script checks the documents check_1 and check_2 and then launches
# createdb.py or start.py
import subprocess
from tkinter import *
import tkinter.messagebox as box
root= Tk()
root.withdraw()
with open('check_1.txt', 'r') as check_1:
for line in check_1:
line = line.strip()
value_1 = int(line)
with open('check_2.txt', 'r') as check_2:
for line in check_2:
line = line.strip()
value_2 = int(line)
if value_1 == 0 and value_2 == 0:
box.showinfo('Libro 1.0', '''
Welcome to the installation of Libro.
I am now creating the database for your catalogue.
This may take a moment.''')
subprocess.call("createdb.py", shell=True)
else:
subprocess.call("start.py", shell=True)
root.mainloop()
It starts, it looks for check_1 and check_2, shows the tkinter showinfo window and then... that's it.
I would be very grateful for any suggestion!! Thanks :)
You would need to freeze all of your scripts, not just the top level one! (create multiple Executable() entries). And then call subprocess to run the frozen executables. If you don't do that you'll end up requiring Python to be installed on the target machine -- and then why freeze any of it! Of course, it might also be helpful to explain why you need to run your code in a subprocess instead of directly.
In the end it looks like it would be easier and more economic to treat my scripts as modules and then import them when needed.
I tried some simplified operations and they seem to work.
For example:
being main.py
from tkinter import *
from modules import from_file
root = Tk()
root.title('Trial window')
btn_1 = Button(root, text='Read from file', command=from_file)
btn_1.grid(row=1, column=1)
and being modules.py
from tkinter import *
def from_file():
ft = open('text.txt', 'r')
string = ''
for line in ft:
line = line.strip()
string = string+line
ft.close()
root2 = Tk()
result = Label(root2, text=string)
result.grid(row=1, column=1)
root2.mainloop()
the script reads and visualize the content of 'text.txt' in the new windows it opens also after being frozen with cx_freeze.
PS The setup.py I used is
from cx_Freeze import setup, Executable
import os.path
PYTHON_INSTALL_DIR = os.path.dirname(os.path.dirname(os.__file__))
os.environ['TCL_LIBRARY'] = os.path.join(PYTHON_INSTALL_DIR, 'tcl', 'tcl8.6')
os.environ['TK_LIBRARY'] = os.path.join(PYTHON_INSTALL_DIR, 'tcl', 'tk8.6')
setup(
name = "Prova",
version = "1.0.0",
options = {"build_exe": {
'packages': ["tkinter"],
'include_files' : [os.path.join(PYTHON_INSTALL_DIR, 'DLLs', 'tk86t.dll'), \
os.path.join(PYTHON_INSTALL_DIR, 'DLLs', 'tcl86t.dll'), 'text.txt'],
'include_msvcr': True,
}},
executables = [Executable("main.py", base="Win32GUI")]
)

Python 3.3 argparse error after compiling to exe with cx_freeze

Trying to compile a Python 3.3 application but running into problems after compiling it to win32 exe using cx_freeze on Windows 7.
Example:
Test.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import argparse
import tkinter.messagebox
cl_parser = argparse.ArgumentParser(
description='Test app Command Line Parameters')
cl_parser.add_argument('--version', action='version', version='%(prog)s 1.0')
cl_parser.add_argument('-s', '--show', help='show simple message box',
action='store_true')
cl_args = cl_parser.parse_args()
if cl_args.show:
tkinter.messagebox.showinfo(title='Simple Message', message='Hi There')
Cx_Freeze Script
import sys
from cx_Freeze import setup, Executable
import shutil
#Remove Build/Dist Folder Before Compile
args = None
print('Removing Old Folders...')
try:
for args in sys.argv:
if args.find('build_exe') != -1:
shutil.rmtree('build', ignore_errors=True)
elif args.find('bdist_msi') != -1:
shutil.rmtree('dist', ignore_errors=True)
except:
pass
finally:
del args
print('Folders Removed')
base = None
if sys.platform == 'win32':
base = 'Win32GUI'
options = {
'build_exe': {
'includes': 'atexit',
'include_msvcr': True
}
}
executables = [
Executable('Test.py', base=base, targetName='Test App.exe')
]
setup(name='Test App',
version='1.0',
description='Test App',
options=options,
executables=executables)
After the successful compile I try a few command line tests on the exe:
Test App.exe -s (This successfully shows the messagebox)
Test App.exe -h or TestApp.exe --version (Fails with AttributeError: 'NoneType' Object has no attribute 'write')
Is there a way that I can make it so that this will work? I am trying to create a PyQt4 application and don't want to have to compile this as a console application.
Your feedback would be appreciated
P.S. Yes I know i've included an example with tkinter but this is for demo purposes only.

Resources