I have a strange problem with a Python3/PyQT5 app, frozen using PyInstaller. On macOS the code runs fine from a CLI. The version frozen using PyInstaller also runs successfully but creates new copies of itself every couple of seconds in what looks like an infinite loop. What could possibly cause this behaviour?
By way of background I have successfully frozen this app on Ubuntu 18.05 and Windows 10 without seeing this problem.
I am running Python 3.7.2 with PyInstaller 3.4 and PyQT5 5.12, on macOS Mojave 10.14.3.
Given the Python code runs perfectly from the CLI and from PyCharm, and the frozen package runs (but with multiple copies of itself) it looks like the problem is somewhere in the freezing process.
The PyInstaller .spec file I am using is here:
# -*- mode: python -*-
import sys
import os
block_cipher = None
def get_assets():
data_files = []
for file_name in os.listdir('/Users/xxx/PycharmProjects/jbrd2/jbrd/assets'):
data_files.append((os.path.join('jbrd/assets', file_name), 'assets'))
data_files.append(('/Users/xxx/PycharmProjects/jbrd2/jbrd/jbrdclasses.py', '.'))
data_files.append(('/Users/xxx/PycharmProjects/jbrd2/jbrd/assets/config.ini', 'assets'))
data_files.append(('/Users/xxx/PycharmProjects/jbrd2/jbrd/python_mysql_connect2.py', '.'))
data_files.append(('/Users/xxx/PycharmProjects/jbrd2/jbrd/python_mysql_dbconfig.py', '.'))
return data_files
a = Analysis(['jbrd/main.py'],
pathex=['/Users/xxx/PycharmProjects/jbrd2'],
binaries=[],
datas=get_assets(),
hiddenimports=['sklearn.naive_bayes','simpleeval', 'configparser', 'mysql.connector'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='jbrd',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=False,
icon='/jbrd.ico')
# Build a .app if on OS X
if sys.platform == 'darwin':
app = BUNDLE(exe,
name='jbrd.app',
icon=None,
bundle_identifier=None,
info_plist={
'NSHighResolutionCapable': 'True'
},
)
I am at a loss as to what could cause an app to run multiple copies of itself. Any suggestion please?
Thanks to some help here 1 I have now narrowed down the cause of this problem.
I can confirm that running a 1-line script:
import sklearn
frozen with PyInstaller causes an infinite loop. Adding an exclusion to the PyInstaller .spec file:
excludes=['sklearn.externals.joblib']
fixes the problem. The underlying problem is that freezing an app that calls joblib causes an infinite loop. You can prove this by freezing a 1-line script
import joblib
and you will see that this causes an infinite loop.
My app does NOT call joblib directly, but uses the machine learning features in sklearn. Unfortunately, sklearn calls joblib and this is what is causing my app to loop when frozen. This appears to be a bug in PyInstaller. Can anyone suggest a workaround?
Related
I have made a Python tool(using PyQt) to work with scanned pdfs which uses tesserocr and imagemagick wand . Both Tessorocr and imagemagick executables I installed at my system and tool is working fine at my system. But now I want to make this tool as single executable to share with people. So that they do not need to install Imagemagick and Tesserocr separately.
I have been searching this problem since days now, but could not get the concrete answer .
Couple of hints I did try .
create SPEC file with dependent binaries
updating environment variable for imagemagick os.environ['MAGICK_HOME'] = './'
But still not able to make single exe.
Binaries path :
Imagemagcik - C:\Program Files\ImageMagick-7.0.10-Q16
TessorOCR - C:\Program Files\Tesseract-OCR
My spec file look like :
a = Analysis(['form.py'],
pathex=['C:\\Users\\usrname\\nlp_repo\\src'],
binaries=[('C:\\Program Files\\ImageMagick-7.0.10-Q16\\magick.exe', 'wand') ,
('C:\\Program Files\\Tesseract-OCR\\tesseract.exe', 'tesserocr')],
datas = [ ('C:/Users/usrname/nlp_repo/src/download.png') ],
hiddenimports=[], hookspath=[],
runtime_hooks=[], excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='form',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='form')
Pyinstaller error-
format_binaries_and_datas
for src_root_path_or_glob, trg_root_dir in binaries_or_datas:
ValueError: too many values to unpack.
I am not sure if any more settings I should include in my Form.SPEC file and how many exe’s should be pass for imagemagcik and tesserocr binaries to make workable executable.
Any steps guide would be helpful.
thank you in advance
I have resolved this issue. To have Python tool running on local system I did following steps
set os.environ['MAGICK_HOME'] = './'
setting hidden import of tessor OCR python pkg dependencies
upgrading setuptools to > 45.0.0.0 (pip install --upgrade setuptools)
a = Analysis(['form.py'],
pathex=['C:\\Users\\usrname\\nlp_repo\\src'],
binaries=[],
datas = [ ('C:\\Users\\usrname\\nlp_repo\\src\\download.png', '.') ],
hiddenimports=['PyTessBaseAPI'],
hookspath=[],
runtime_hooks=[],
excludes=[],
.....
binaries can be bundle as standalone(by setting binaries in spec files) and then target system user can install it separately and then can easily use Python tool generated using Pyinstaller.
Jsh
NOTE:
I am relatively new to Data Science / coding, so please let me know if I'm missing something fundamental.
Question
What is the suggested procedure for implementing a machine learning model in Microsoft excel?
My strategy for doing this has been the following:
1.) Vectorize my text data for training
2.) Train a logistic regression model
3.) Pickle trained model and fit vectorizer
4.) Load model/vectorizer into separate python file
5.) Convert python file into .exe file using PyInstaller
6.) Call python.exe file in Excel VBA, classifying text data as required
This procedure works, and I can call the model to classify text. However, a big problem is how long it takes to execute the .exe file. The python code that loads the pickled data only takes 10-20 seconds to execute. On the other hand, the .exe derived from this code takes at least 5 minute to execute given just a single string entry to classify.
The .exe file is also very large at 305 MB. This is probably due to installation of the sklearn modules needed to load the model and vectorizer. I am not sure if there is a lighter weight solution for this. I am certain there is a better alternative for a simple logistic regression model.
Ultimately, I am looking for any suggestions on how I may run my machine learning models in excel efficiently.
Sample of Python Code
Generate Logistic Regression Model and Pickle
from sklearn.externals import joblib
''' https://www.geeksforgeeks.org/saving-a-machine-learning-model/ '''
path = "N:\\Drive_Location\\Pickles"
model_string = "eSOMS_Logistic_Regression_Model.pkl"
vectorizer_transformer_string = "eSOMS_Logistic_Regression_vectorizer_transformer.pkl"
model_full_path = path +'\\' + model_string
vectorizer_full_path = path +'\\' + vectorizer_transformer_string
''' Save the model as a pickle in a file '''
joblib.dump(classifier, model_full_path)
joblib.dump(vectorizer.transform, vectorizer_full_path)
logistic_regression_from_joblib = joblib.load(model_full_path)
vectorizer_transformer_from_joblib = joblib.load(vectorizer_full_path)
Load the Pickled File into Separate Python File
from sklearn.externals import joblib
path = "N:\\Drive_Location\\Pickles"
model_string = "eSOMS_Logistic_Regression_Model.pkl"
vectorizer_transformer_string = "eSOMS_Logistic_Regression_vectorizer_transformer.pkl"
model_full_path = path +'\\' + model_string
vectorizer_full_path = path +'\\' + vectorizer_transformer_string
logistic_regression_from_joblib = joblib.load(model_full_path)
vectorizer_transformer_from_joblib = joblib.load(vectorizer_full_path)
import sys
argument = sys.argv[1]
def myfunction(string):
print (logistic_regression_from_joblib.predict(vectorizer_transformer_from_joblib([string])))
myfunction(argument)
Create .Exe File in Command Prompt using PyInstaller
In command prompt, I created an executable file from the .py file.
pyinstaller --onefile eSOMS_MS_Executable.spec
NOTE: I adjusted the .spec file fed to PyInstaller to up the recursion limit and to specify numerous hidden imports:
# -*- mode: python -*-
import sys
sys.setrecursionlimit(5000)
block_cipher = None
a = Analysis(['eSOMS_MS_Executable.py'],
pathex=['C:\\Users\\Username\\Desktop\\Python\\eSOMS'],
binaries=[],
datas=[],
hiddenimports = ['sklearn.linear_model', 'sklearn.utils.sparsetools._graph_validation', 'sklearn.utils.sparsetools._graph_tools', 'sklearn.neighbors.typedefs', 'sklearn.feature_extraction'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='eSOMS_MS_Executable',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True )
Code works, but it's just too slow to be practical.
After discussing with IT friends on mine, they directed me to this post: App created with PyInstaller has a slow startup
I learned that executable created from --onefile can be very large / slow. I changed to --onedir and the executable became much smaller, and now is only 15 MB (big improvement vs. 305 MB).
pyinstaller --onedir eSOMS_MS_Executable.spec
Thanks guys!
Good day,
While trying to make a standalone (--onefile) exe with Pyinstaller, I ran into a strange error. The execution does not find two .dll even though they were properly packed into the exe and are properly placed into a temp/_MEIPASSxxxxx folder at runtime, as the screenshots below show. This does not happen on the development computer.
.
I added my data (icon & readme) in this folder too thanks to this thread and the program has no problem with finding them, so the issue is only with binaries and not data apparently. Also there is no problem at all in --onedir mode, only --onefile.
Here is my .spec:
# -*- mode: python -*-
block_cipher = None
import sys
sys.setrecursionlimit(5000)
added_binaries = [
("python36.dll", "."),
("api-ms-win-crt-runtime-l1-1-0.dll", ".")
]
added_data = [
("PilotageIcon.png", "."),
("PilotageREADME.pdf", ".")
]
a = Analysis(['Pilotage.py'],
pathex=['C:\\Users\\local-adm\\Desktop\\Lucas\\Keithley\\2018 07 18'],
binaries=added_binaries,
datas=added_data,
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='Pilotage',
debug=True,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True,
icon='PilotageIcon.ico'
)
I read many other threads here but they do not match my situation as clearly the files are there, so I don't understand why the errors. Thank you in advance.
Python version: 3.6.3
Pyinstaller version: 3.3.1
A possible workaround, not a fix, is to install Visual C++ Redistributable for Visual Studio 2015 on the machine. Hopefully that helps someone. Credit.
It's not exactly ideal because it adds more steps to follow for the final user.
I'm quite new to programming with Python so I might be missing something quite obvious here.
I've done a small program in Python which runs ok, but I wanted to share it with my colleagues so I thought of doing a standalone version of it, I've tried using Pyinstaller as it seems the easiest and more straightforward to use, but the exe file generated doesn't seem to work so far.
So I have my program divided in a main file called "main.py" and 3 extra functions that I import in main from other python files in the same folder. It uses one excel spreadsheet and one .ico image for the logo in tkinter.
My first try has been using just the command:
pyinstaller main.py
It generates the exe file, but when executing it it just opens a black window and nothing else happens.
I was reading the documentation for pyinstaller and thought I might need to add the other python files too and the extra files. For the excel and the image I didn't manage to understand how I need to write it, I tried to add the following as options, but it gave me back an error
--add-data <logo.ico> --add-data <WAE_SW_NameCnvtn.xlsx>
Without those, I managed to generate again the exe file using the following command:
pyinstaller --windowed --name=NameCnvtn main.py read_description.py read_quantity.py word_finder.py
This seemed to add quite a few more files to the dist folder, and when I execute the exe file instead of the black window it says "failed to execute the script main", which I assume referes to my main.py file.
In case it helps, I developed it using python 3.6.4 and it uses tkinter and openpyxl.
Thanks for the help.
I managed to successfully run the exe today. I manually added in the spec file the excel and the ico under the data line, compiled it again and it was running correctly after that. here is the spec file in case someone has the same problem as I had.
# -*- mode: python -*-
block_cipher = None
a = Analysis(['main.py', 'read_description.py', 'read_quantity.py', 'word_finder.py'],
pathex=['C:\\Users\\j.berjano\\Desktop\\app_test'],
binaries=[],
datas=[('WAE_SW_NameCnvtn.xlsx','.'),('logo.ico','.')],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
exclude_binaries=True,
name='NameCnvtn',
debug=False,
strip=False,
upx=True,
console=False,
icon='logo.ico')
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
name='NameCnvtn')
I have a PyQt application that I am trying to compile into one exe using PyInstaller 3.1. It does compile into an exe, however, the GUI doesn't show up upon launch (clicking the exe). One of the compilation message is:
ImportError: No module named site 5576 ERROR: Cannot find PyQt4 plugin
directories 5576 WARNING: Hidden import 'PyQT4._qt' not found
(probably old hook)
I think this is why the UI wouldn't show up. I've already add all of the paths that has PyQt4 in the pathex attribute for the Analysis object to no avail. Below is my spec file:
# -*- mode: python -*-
block_cipher = None
a = Analysis(['main32.py'],
pathex=['C:/OSGeo4W/apps/Python27/Lib/site-packages/PyQt4','./src'],
binaries=None,
datas=None,
hiddenimports=[],
hookspath=None,
runtime_hooks=None,
excludes=None,
win_no_prefer_redirects=None,
win_private_assemblies=None,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='MyExe',
debug=False,
strip=None,
upx=True,
console=False )
Any ideas what's next to try?
I had the same error, and I did solve by doing this:
Uninstalling PyQt4("pip uninstall PyQt4" on CMD);
Deleting the "PyQt4" folder at "...PythonX\Lib\site-packages\";
Downloading the ".whl" package from this site: Python Expansion Packages, YOU NEED TO DOWNLOAD THE RIGHT WHL, look the name of the file, since Im using the Python3.5, the file "PyQt4‑4.11.4‑cp35‑none‑win32.whl" worked for me!
Installing again, opening the CMD at the file's location and typing:
"pip install PyQt4‑4.11.4‑cp35‑none‑win32.whl". Of course you'll type your file's name.
Note: Only by uninstalling, I could compile my script. But if you are really using PyQt4 code, you gonna need to install it again.
Please, thumbs up if it worked for you too! ;)