I'm working on packaging a machine learning codebase that has many dependencies on modern machine learning libraries (sklearn, xgboost, tensorflow, pandas, and the like). We are trying to bundle the codebase into a single folder containing an executable and additional binaries needed for running (shared libraries). To do this, we're trying to use PyInstaller.
I am able to successfully create an executable on MacOS (mojave 10.14.6) and CentOS 7.7. I'm using conda (4.7.12) to help manage virtual environments, and Python 3.6.6. On both platforms, I can run the code from source (i.e., not the exe created by PyInstaller) and everything runs normally. Running the executable on MacOS works normally. However, running the executable on CentOS ends with a seg fault (it's unclear where this is actually happening).
Here is the output of build process on CentOS:
54 INFO: PyInstaller: 3.5
54 INFO: Python: 3.6.6
55 INFO: Platform: Linux-3.10.0-1062.1.2.el7.x86_64-x86_64-with-centos-7.7.1908-Core
58 INFO: UPX is not available.
59 INFO: Removing temporary files and cleaning cache in /root/.cache/pyinstaller
60 INFO: Extending PYTHONPATH with paths
['/home/jloy/amb-rte', '/home/jloy/amb-rte']
60 INFO: checking Analysis
61 INFO: Building Analysis because Analysis-00.toc is non existent
61 INFO: Initializing module dependency graph...
64 INFO: Initializing module graph hooks...
66 INFO: Analyzing base_library.zip ...
3879 INFO: Analyzing hidden import 'amb_pacman.settings.rte'
4869 INFO: Analyzing hidden import 'amb.settings.rte'
5161 INFO: Processing pre-find module path hook distutils
6708 INFO: Processing pre-safe import module hook six.moves
7724 INFO: Processing pre-safe import module hook setuptools.extern.six.moves
8357 INFO: Processing pre-find module path hook site
8358 INFO: site: retargeting to fake-dir '/root/miniconda3/envs/rte/lib/python3.6/site-packages/PyInstaller/fake-modules'
37973 INFO: Analyzing hidden import 'amb.evolve.neat.core.indexer'
38049 INFO: Analyzing hidden import 'amb_utils.settings.rte'
39648 INFO: Analyzing hidden import 'sklearn.utils._cython_blas'
39648 INFO: Analyzing hidden import 'sklearn.neighbors.typedefs'
39648 INFO: Analyzing hidden import 'sklearn.neighbors.quad_tree'
39649 INFO: Analyzing hidden import 'sklearn.tree._utils'
39734 INFO: Analyzing hidden import 'multiprocessing.forking'
39734 ERROR: Hidden import 'multiprocessing.forking' not found
39734 INFO: Analyzing hidden import 'multiprocessing.freeze_support'
39734 ERROR: Hidden import 'multiprocessing.freeze_support' not found
39734 INFO: Analyzing hidden import 'numpy.type'
39735 ERROR: Hidden import 'numpy.type' not found
39735 INFO: running Analysis Analysis-00.toc
39762 INFO: Caching module hooks...
39768 INFO: Analyzing /home/jloy/amb-rte/runtime_engine.py
51355 INFO: Processing pre-safe import module hook win32com
68391 INFO: Processing pre-safe import module hook urllib3.packages.six.moves
101910 INFO: Loading module hooks...
101911 INFO: Loading module hook "hook-IPython.py"...
101926 INFO: Excluding import 'matplotlib'
101941 INFO: Removing import of matplotlib._pylab_helpers from module IPython.core.pylabtools
101941 INFO: Removing import of matplotlib.pyplot from module IPython.core.pylabtools
101941 INFO: Removing import of matplotlib.figure from module IPython.core.pylabtools
101941 INFO: Removing import of matplotlib from module IPython.core.pylabtools
101943 INFO: Excluding import 'PySide'
101957 INFO: Excluding import 'PyQt5'
101972 INFO: Excluding import 'PyQt4'
101986 INFO: Excluding import 'gtk'
102000 INFO: Loading module hook "hook-PIL.Image.py"...
102458 INFO: Loading module hook "hook-PIL.SpiderImagePlugin.py"...
102483 INFO: Import to be excluded not found: 'FixTk'
102484 INFO: Excluding import 'tkinter'
102496 INFO: Loading module hook "hook-PIL.py"...
102506 INFO: Excluding import 'PySide'
102521 INFO: Removing import of PySide from module PIL.ImageQt
102522 INFO: Excluding import 'PyQt5'
102534 INFO: Removing import of PyQt5.QtCore from module PIL.ImageQt
102534 INFO: Removing import of PyQt5.QtGui from module PIL.ImageQt
102535 INFO: Excluding import 'PyQt4'
102547 INFO: Removing import of PyQt4 from module PIL.ImageQt
102548 INFO: Import to be excluded not found: 'FixTk'
102548 INFO: Excluding import 'tkinter'
102559 INFO: Removing import of tkinter from module PIL.ImageTk
102559 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
102708 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
102832 INFO: Loading module hook "hook-PyQt5.QtWidgets.py"...
103067 INFO: Loading module hook "hook-PyQt5.py"...
103489 WARNING: Hidden import "PyQt5.sip" not found!
103489 INFO: Loading module hook "hook-_tkinter.py"...
103656 INFO: checking Tree
103657 INFO: Building Tree because Tree-00.toc is non existent
103657 INFO: Building Tree Tree-00.toc
103665 INFO: checking Tree
103666 INFO: Building Tree because Tree-01.toc is non existent
103666 INFO: Building Tree Tree-01.toc
103670 INFO: Loading module hook "hook-certifi.py"...
103672 INFO: Loading module hook "hook-cryptography.py"...
103978 INFO: Loading module hook "hook-distutils.py"...
103979 INFO: Loading module hook "hook-encodings.py"...
104077 INFO: Loading module hook "hook-h5py.py"...
104079 INFO: Loading module hook "hook-importlib_metadata.py"...
104082 INFO: Loading module hook "hook-jedi.py"...
104127 INFO: Loading module hook "hook-jinja2.py"...
104157 INFO: Loading module hook "hook-lib2to3.py"...
104159 INFO: Loading module hook "hook-llvmlite.py"...
104161 INFO: Loading module hook "hook-matplotlib.backends.py"...
104933 INFO: Matplotlib backend "GTK": ignored
Gtk* backend requires pygtk to be installed.
105583 INFO: Matplotlib backend "GTKAgg": ignored
Gtk* backend requires pygtk to be installed.
106039 INFO: Matplotlib backend "GTKCairo": ignored
No module named 'gtk'
106663 INFO: Matplotlib backend "GTK3Agg": ignored
The Gtk3 backend requires PyGObject or pgi
107050 INFO: Matplotlib backend "GTK3Cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
107625 INFO: Matplotlib backend "MacOSX": ignored
cannot import name '_macosx'
/root/miniconda3/envs/rte/lib/python3.6/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
"You should import from ipykernel or jupyter_client instead.", ShimWarning)
108462 INFO: Matplotlib backend "nbAgg": ignored
No module named 'ipykernel'
109134 INFO: Matplotlib backend "Qt4Agg": added
109487 INFO: Matplotlib backend "Qt4Cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
110150 INFO: Matplotlib backend "Qt5Agg": added
110519 INFO: Matplotlib backend "Qt5Cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
111089 INFO: Matplotlib backend "TkAgg": added
111452 INFO: Matplotlib backend "TkCairo": ignored
cairo backend requires that cairocffi or pycairo is installed
112115 INFO: Matplotlib backend "WebAgg": added
112741 INFO: Matplotlib backend "WX": ignored
Matplotlib backend_wx and backend_wxagg require wxPython>=2.9
113172 INFO: Matplotlib backend "WXAgg": ignored
No module named 'wx'
113603 INFO: Matplotlib backend "WXCairo": ignored
No module named 'wx'
114279 INFO: Matplotlib backend "agg": added
114704 INFO: Matplotlib backend "cairo": ignored
cairo backend requires that cairocffi or pycairo is installed
115090 INFO: Matplotlib backend "gdk": ignored
No module named 'gobject'
115640 INFO: Matplotlib backend "pdf": added
116288 INFO: Matplotlib backend "pgf": added
116838 INFO: Matplotlib backend "ps": added
117424 INFO: Matplotlib backend "svg": added
117967 INFO: Matplotlib backend "template": added
118562 INFO: Loading module hook "hook-matplotlib.py"...
118991 INFO: Loading module hook "hook-numba.py"...
119014 INFO: Excluding import 'IPython'
119033 INFO: Removing import of IPython.paths from module numba.caching
119033 INFO: Removing import of IPython.utils.path from module numba.caching
119040 INFO: Excluding import 'scipy'
119057 INFO: Removing import of scipy.linalg.cython_blas from module numba.targets.linalg
119057 INFO: Removing import of scipy.linalg.cython_lapack from module numba.targets.linalg
119063 INFO: Loading module hook "hook-numpy.core.py"...
119229 INFO: MKL libraries found when importing numpy. Adding MKL to binaries
119230 INFO: Loading module hook "hook-numpy.py"...
119231 INFO: Loading module hook "hook-pandas.py"...
120598 INFO: Loading module hook "hook-patsy.py"...
120600 INFO: Loading module hook "hook-pkg_resources.py"...
121246 INFO: Loading module hook "hook-pycparser.py"...
121247 INFO: Loading module hook "hook-pydoc.py"...
121247 INFO: Loading module hook "hook-pygments.py"...
123779 INFO: Loading module hook "hook-pytest.py"...
124808 INFO: Loading module hook "hook-pytz.py"...
124826 INFO: Loading module hook "hook-scipy.io.matlab.py"...
124827 INFO: Loading module hook "hook-scipy.linalg.py"...
124827 INFO: Loading module hook "hook-scipy.py"...
124828 INFO: Loading module hook "hook-scipy.sparse.csgraph.py"...
124831 INFO: Loading module hook "hook-scipy.special._ellip_harm_2.py"...
124831 INFO: Loading module hook "hook-scipy.special._ufuncs.py"...
124832 INFO: Loading module hook "hook-setuptools.py"...
125823 INFO: Loading module hook "hook-shelve.py"...
125824 INFO: Loading module hook "hook-sklearn.metrics.cluster.py"...
125851 WARNING: Hidden import "sklearn.utils.sparsetools._graph_validation" not found!
125851 WARNING: Hidden import "sklearn.utils.sparsetools._graph_tools" not found!
125852 INFO: Loading module hook "hook-sqlite3.py"...
125944 INFO: Loading module hook "hook-sysconfig.py"...
125956 INFO: Loading module hook "hook-torch.py"...
125987 INFO: Loading module hook "hook-xml.dom.domreg.py"...
125987 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
125988 INFO: Loading module hook "hook-xml.py"...
125988 INFO: Loading module hook "hook-xgboost.py"...
126102 INFO: Determining a mapping of distributions to packages...
157007 INFO: Packages required by xgboost:
['numpy', 'scipy']
157271 INFO: checking Tree
157272 INFO: Building Tree because Tree-02.toc is non existent
157272 INFO: Building Tree Tree-02.toc
157273 INFO: checking Tree
157273 INFO: Building Tree because Tree-03.toc is non existent
157273 INFO: Building Tree Tree-03.toc
157273 INFO: checking Tree
157273 INFO: Building Tree because Tree-04.toc is non existent
157273 INFO: Building Tree Tree-04.toc
157274 INFO: checking Tree
157274 INFO: Building Tree because Tree-05.toc is non existent
157274 INFO: Building Tree Tree-05.toc
157275 INFO: checking Tree
157275 INFO: Building Tree because Tree-06.toc is non existent
157275 INFO: Building Tree Tree-06.toc
157275 INFO: checking Tree
157275 INFO: Building Tree because Tree-07.toc is non existent
157275 INFO: Building Tree Tree-07.toc
157276 INFO: checking Tree
157276 INFO: Building Tree because Tree-08.toc is non existent
157276 INFO: Building Tree Tree-08.toc
157277 INFO: checking Tree
157277 INFO: Building Tree because Tree-09.toc is non existent
157277 INFO: Building Tree Tree-09.toc
157277 INFO: checking Tree
157277 INFO: Building Tree because Tree-10.toc is non existent
157277 INFO: Building Tree Tree-10.toc
157278 INFO: checking Tree
157278 INFO: Building Tree because Tree-11.toc is non existent
157279 INFO: Building Tree Tree-11.toc
157279 INFO: checking Tree
157279 INFO: Building Tree because Tree-12.toc is non existent
157279 INFO: Building Tree Tree-12.toc
157279 INFO: checking Tree
157280 INFO: Building Tree because Tree-13.toc is non existent
157280 INFO: Building Tree Tree-13.toc
157280 INFO: checking Tree
157280 INFO: Building Tree because Tree-14.toc is non existent
157280 INFO: Building Tree Tree-14.toc
157281 INFO: checking Tree
157282 INFO: Building Tree because Tree-15.toc is non existent
157282 INFO: Building Tree Tree-15.toc
157282 INFO: checking Tree
157282 INFO: Building Tree because Tree-16.toc is non existent
157282 INFO: Building Tree Tree-16.toc
157284 INFO: checking Tree
157284 INFO: Building Tree because Tree-17.toc is non existent
157284 INFO: Building Tree Tree-17.toc
157285 INFO: checking Tree
157285 INFO: Building Tree because Tree-18.toc is non existent
157285 INFO: Building Tree Tree-18.toc
157307 INFO: checking Tree
157307 INFO: Building Tree because Tree-19.toc is non existent
157307 INFO: Building Tree Tree-19.toc
157315 INFO: checking Tree
157316 INFO: Building Tree because Tree-20.toc is non existent
157316 INFO: Building Tree Tree-20.toc
157316 INFO: Looking for ctypes DLLs
157838 WARNING: library user32 required via ctypes not found
159961 INFO: Analyzing run-time hooks ...
160067 INFO: Including run-time hook 'pyi_rth_traitlets.py'
160073 INFO: Including run-time hook 'pyi_rth__tkinter.py'
160078 INFO: Including run-time hook 'pyi_rth_mplconfig.py'
160082 INFO: Including run-time hook 'pyi_rth_mpldata.py'
160085 INFO: Including run-time hook 'pyi_rth_pyqt5.py'
160088 INFO: Including run-time hook 'pyi_rth_pkgres.py'
160092 INFO: Including run-time hook 'pyi_rth_certifi.py'
160096 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
160297 INFO: Looking for dynamic libraries
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/cryptography-2.5-py3.6-linux-x86_64.egg/cryptography/hazmat/bindings/_constant_time.abi3.so'
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/cryptography-2.5-py3.6-linux-x86_64.egg/cryptography/hazmat/bindings/_openssl.abi3.so'
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/cryptography-2.5-py3.6-linux-x86_64.egg/cryptography/hazmat/bindings/_padding.abi3.so'
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/lib/python3.6/site-packages/coverage-5.0a8-py3.6-linux-x86_64.egg/coverage/tracer.cpython-36m-x86_64-linux-gnu.so'
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/tbbpool.cpython-36m-x86_64-linux-gnu.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/tbbpool.cpython-36m-x86_64-linux-gnu.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/tbbpool.cpython-36m-x86_64-linux-gnu.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
ldd: warning: you do not have execution permission for `/root/miniconda3/envs/rte/plugins/platforms/../../lib/libgcc_s.so.1'
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2: /lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
/root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /root/miniconda3/envs/rte/lib/python3.6/site-packages/numba/npyufunc/../.libs/libtbb-52872150.so.2)
173700 INFO: Looking for eggs
173701 INFO: Using Python library /root/miniconda3/envs/rte/lib/python3.6/site-packages/PIL/../../../libpython3.6m.so.1.0
173911 INFO: Warnings written to /home/jloy/amb-rte/pydist/build/runtime_engine/warn-runtime_engine.txt
174792 INFO: Graph cross-reference written to /home/jloy/amb-rte/pydist/build/runtime_engine/xref-runtime_engine.html
175119 INFO: Appending 'datas' from .spec
175132 INFO: checking PYZ
175132 INFO: Building PYZ because PYZ-00.toc is non existent
175132 INFO: Building PYZ (ZlibArchive) /home/jloy/amb-rte/pydist/build/runtime_engine/PYZ-00.pyz
188154 INFO: Building PYZ (ZlibArchive) /home/jloy/amb-rte/pydist/build/runtime_engine/PYZ-00.pyz completed successfully.
188323 INFO: checking PKG
188324 INFO: Building PKG because PKG-00.toc is non existent
188325 INFO: Building PKG (CArchive) PKG-00.pkg
188418 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
188420 INFO: Bootloader /root/miniconda3/envs/rte/lib/python3.6/site-packages/PyInstaller/bootloader/Linux-64bit/run_d
188420 INFO: checking EXE
188420 INFO: Building EXE because EXE-00.toc is non existent
188420 INFO: Building EXE from EXE-00.toc
188420 INFO: Appending archive to ELF section in EXE /home/jloy/amb-rte/pydist/build/runtime_engine/runtime_engine
188523 INFO: Building EXE from EXE-00.toc completed successfully.
188535 INFO: checking COLLECT
188536 INFO: Building COLLECT because COLLECT-00.toc is non existent
188536 INFO: Building COLLECT COLLECT-00.toc
193501 INFO: Building COLLECT COLLECT-00.toc completed successfully.
The ldd warnings and the messages about GLIBCXX do not happen for MacOS. Either way, the process creates an executable that is runnable.
I'm very perplexed as to how it's possible I can run the code unpackaged with no problems, and at the same time, the executable fails with a seg fault. Especially since I'm running on the exact same machine that created the executable. That is, I'm essentially doing this:
$ pyinstaller mySpec.spec
$ ./dist/myExe
Any help with this issue would be greatly appreciated. I would be happy to provide additional information as well.
Thanks in advance.
UPDATE:
I was able to get the executable running on CentOS 7.7. There ended up being an issue with which BLAS library numpy used. When I forced conda to use the openblas flavor (instead of MKL), everything worked.
I was able to figure this out by trying the same procedure on CentOS 8. When running the executable there I got a more descriptive error, instead of just a seg fault. The executable still crashed, but it was because it could not find the symbol __kmpc_global_thread_num in libmkl_intel_thread.so. I started googling around and was able to find someone with a similar issue who solved it by forcing openblas over MKL.
The question still remains though: why would running from source work, but running the executable not?
Try setting up a new conda environment with Python 3.5 (not 3.6 or above) and compile from it. It has worked for me in centOS 6.
You may need to add non-standard libraries (like numpy) as hidden imports for pyinstaller.
Hope it helps
Related
I'm trying to create a executable binary for a python codebase using pyinstaller. When running pyinstaller on Linux it creates the binary, but the binary fails to run because pyinstaller doesn't properly include QtMultimedia. It worked fine when I used the same procedure on Windows.
The following code reproduces the issue:
from PyQt5 import QtMultimedia, QtWidgets
import sys
class MainWindow(QtWidgets.QMainWindow):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
test_sound = QtMultimedia.QSound("test_sound.wav")
if __name__ == "__main__":
app = QtWidgets.QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()
Running Pyinstaller
.../install_tester$ pyinstaller test.py --onefile --hidden-import=PyQt5.QtMultimedia
875 INFO: PyInstaller: 4.8
875 INFO: Python: 3.8.8 (conda)
896 INFO: Platform: Linux-5.13.0-37-generic-x86_64-with-glibc2.10
896 INFO: wrote ...install_tester/test.spec
899 INFO: UPX is not available.
900 INFO: Extending PYTHONPATH with paths
['...install_tester']
1093 INFO: checking Analysis
1096 INFO: Building because hiddenimports changed
1096 INFO: Initializing module dependency graph...
1097 INFO: Caching module graph hooks...
1103 INFO: Analyzing base_library.zip ...
2980 INFO: Processing pre-find module path hook distutils from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2980 INFO: distutils: retargeting to non-venv dir '~anaconda3/lib/python3.8'
5648 INFO: Caching module dependency graph...
5749 INFO: running Analysis Analysis-00.toc
5771 INFO: Analyzing ...install_tester/test.py
5814 INFO: Processing module hooks...
5814 INFO: Loading module hook 'hook-difflib.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5816 INFO: Loading module hook 'hook-multiprocessing.util.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5816 INFO: Loading module hook 'hook-distutils.util.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5817 INFO: Loading module hook 'hook-pickle.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5818 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5819 INFO: Loading module hook 'hook-encodings.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5857 INFO: Loading module hook 'hook-heapq.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
5858 INFO: Loading module hook 'hook-PyQt5.QtWidgets.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
6157 INFO: Loading module hook 'hook-_tkinter.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
6218 INFO: checking Tree
6222 INFO: checking Tree
6258 INFO: checking Tree
6259 INFO: Loading module hook 'hook-distutils.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
6265 INFO: Loading module hook 'hook-sysconfig.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
6265 INFO: Loading module hook 'hook-lib2to3.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
6281 INFO: Loading module hook 'hook-xml.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
6345 INFO: Loading module hook 'hook-PyQt5.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
7478 WARNING: Hidden import "PyQt5.sip" not found!
7478 INFO: Loading module hook 'hook-PyQt5.QtGui.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
7531 INFO: Loading module hook 'hook-PyQt5.QtCore.py' from '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks'...
7566 INFO: Looking for ctypes DLLs
7602 INFO: Analyzing run-time hooks ...
7606 INFO: Including run-time hook '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_subprocess.py'
7607 INFO: Including run-time hook '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
7608 INFO: Including run-time hook '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
7610 INFO: Including run-time hook '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
7611 INFO: Including run-time hook '~anaconda3/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pyqt5.py'
7615 INFO: Looking for dynamic libraries
8880 INFO: Looking for eggs
8880 INFO: Python library not in binary dependencies. Doing additional searching...
8892 INFO: Using Python library ~anaconda3/lib/libpython3.8.so
8898 INFO: Warnings written to ...install_tester/build/test/warn-test.txt
8919 INFO: Graph cross-reference written to ...install_tester/build/test/xref-test.html
8926 INFO: checking PYZ
8928 INFO: Building because toc changed
8928 INFO: Building PYZ (ZlibArchive) ...install_tester/build/test/PYZ-00.pyz
9198 INFO: Building PYZ (ZlibArchive) ...install_tester/build/test/PYZ-00.pyz completed successfully.
9202 INFO: checking PKG
9202 INFO: Building because toc changed
9202 INFO: Building PKG (CArchive) test.pkg
28025 INFO: Building PKG (CArchive) test.pkg completed successfully.
28028 INFO: Bootloader ~anaconda3/lib/python3.8/site-packages/PyInstaller/bootloader/Linux-64bit-intel/run
28029 INFO: checking EXE
28029 INFO: Building because name changed
28029 INFO: Building EXE from EXE-00.toc
WARNING: The output directory "...install_tester/dist/test" and ALL ITS CONTENTS will be REMOVED! Continue? (y/N)y
On your own risk, you can use the option `--noconfirm` to get rid of this question.
33016 INFO: Removing dir ...install_tester/dist/test
33038 INFO: Copying bootloader EXE to ...install_tester/dist/test
33039 INFO: Appending PKG archive to custom ELF section in EXE
33419 INFO: Building EXE from EXE-00.toc completed successfully.
Running the binary file:
...install_tester/dist$ ./test
Traceback (most recent call last):
File "test.py", line 1, in <module>
ImportError: cannot import name 'QtMultimedia' from 'PyQt5' (/tmp/_MEIVI98hn/PyQt5/__init__.pyc)
[33541] Failed to execute script 'test' due to unhandled exception!
I've tried with and without `--hidden-import=PyQt5.QtMultimedia`
I've tried it with and without the --hidden-import flag.
If I discard the --onefile flag, then the folder dist/PyQt5 contains:
install_tester/dist/test/PyQt5$ ls
Qt QtCore.so QtGui.so QtWidgets.so
Noticeably missing QtMultimedia.so, if I try to copy this file from the virtual environment folder, the error message changes to this:
install_tester/dist/test$ ./test
Traceback (most recent call last):
File "test.py", line 1, in <module>
ImportError: .../install_tester/dist/test/libQt5Core.so.5: version `Qt_5.15' not found (required by .../install_tester/dist/test/PyQt5/QtMultimedia.abi3.so)
[35195] Failed to execute script 'test' due to unhandled exception!
I know there have been other questions here regarding hidden imports, following some comments from those posts I've also tried changing test.spec to include this line:
pathex=['.../install_tester/venv/lib/python3.8/site-packages/PyQt5/QtMultimedia.abi3.so'],
But that also doesn't work.
Edit 24-03-22: This issue is caused by using virtual python environments. The exact problem with my virtual environment I'm not entirely sure of, but I successfully compiled a linux binary by creating a Ubuntu virtual machine (fresh install, and yes... on an Ubuntu host) then installing the python libraries with pip, no conda or other virtual environments. I then ran pyinstaller and copied the binary back to my host machine... it works perfectly!
I used to build using PyInstaller and everything was working perfectly. I don't know what has changed. I had a pre-configured VM that I used to compile and now it doesn't work as expected.
When I build now, the GUI looks like GTK (very old looking GUI) for some reason.
I've made a VERY simple PyQt5 app and made a clean VM with Python 3.6.4, PyInstaller, PyQt5, and six.
I've tried on macOS 10.10, 10.13, Windows 10, Windows 10 VM.
I've tried --hidden-import=PyQt5
I've tried everything again using Python 3.6.3 and PyInstaller 3.3
I've tried everything again using Python 3.6.3 and PyInstaller dev
I'm running out of things to test.
Here is the simple PyQt5 app sample app:
PROBLEM.zip
Here is the build output:
C:\Users\Admin\Desktop\PROBLEM>C:\Python\python -m PyInstaller --paths C:\Python\Lib\site-packages\PyQt5\Qt\bin --add-binary *.dll;. --clean --noupx --name "TEST" main.py
250 INFO: PyInstaller: 3.4.dev0+1652c1968
265 INFO: Python: 3.6.4
265 INFO: Platform: Windows-10-10.0.16299-SP0
265 INFO: wrote C:\Users\Admin\Desktop\PROBLEM\TEST.spec
265 INFO: UPX is not available.
265 INFO: Removing temporary files and cleaning cache in C:\Users\Admin\AppData\Roaming\pyinstaller
296 INFO: Extending PYTHONPATH with paths
['C:\\Users\\Admin\\Desktop\\PROBLEM',
'C:\\Python\\Lib\\site-packages\\PyQt5\\Qt\\bin',
'C:\\Users\\Admin\\Desktop\\PROBLEM']
296 INFO: checking Analysis
312 INFO: Building Analysis because out00-Analysis.toc is non existent
312 INFO: Initializing module dependency graph...
312 INFO: Initializing module graph hooks...
312 INFO: Analyzing base_library.zip ...
3421 INFO: running Analysis out00-Analysis.toc
3437 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by C:\Python\python.exe
3609 INFO: Caching module hooks...
3609 INFO: Analyzing C:\Users\Admin\Desktop\PROBLEM\main.py
3640 INFO: Loading module hooks...
3640 INFO: Loading module hook "hook-encodings.py"...
3718 INFO: Loading module hook "hook-pydoc.py"...
3718 INFO: Loading module hook "hook-PyQt5.py"...
3718 INFO: Loading module hook "hook-PyQt5.Qt.py"...
3718 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
3782 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
4125 INFO: Loading module hook "hook-PyQt5.QtPrintSupport.py"...
4171 INFO: Loading module hook "hook-PyQt5.QtWidgets.py"...
4187 INFO: Loading module hook "hook-xml.py"...
4422 INFO: Looking for ctypes DLLs
4437 INFO: Analyzing run-time hooks ...
4437 INFO: Including run-time hook 'pyi_rth_qt5.py'
4437 INFO: Looking for dynamic libraries
5187 INFO: Looking for eggs
5187 INFO: Using Python library C:\Python\python36.dll
5187 INFO: Found binding redirects:
[]
5203 INFO: Warnings written to C:\Users\Admin\Desktop\PROBLEM\build\TEST\warnTEST.txt
5250 INFO: Graph cross-reference written to C:\Users\Admin\Desktop\PROBLEM\build\TEST\xref-TEST.html
5281 INFO: Appending 'binaries' from .spec
5328 INFO: checking PYZ
5328 INFO: Building PYZ because out00-PYZ.toc is non existent
5328 INFO: Building PYZ (ZlibArchive) C:\Users\Admin\Desktop\PROBLEM\build\TEST\out00-PYZ.pyz
5890 INFO: Building PYZ (ZlibArchive) C:\Users\Admin\Desktop\PROBLEM\build\TEST\out00-PYZ.pyz completed successfully.
5906 INFO: checking PKG
5906 INFO: Building PKG because out00-PKG.toc is non existent
5906 INFO: Building PKG (CArchive) out00-PKG.pkg
5937 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
5953 INFO: Bootloader C:\Python\lib\site-packages\PyInstaller\bootloader\Windows-32bit\run.exe
5953 INFO: checking EXE
5953 INFO: Building EXE because out00-EXE.toc is non existent
5953 INFO: Building EXE from out00-EXE.toc
5953 INFO: Appending archive to EXE C:\Users\Admin\Desktop\PROBLEM\build\TEST\TEST.exe
5984 INFO: Building EXE from out00-EXE.toc completed successfully.
6000 INFO: checking COLLECT
6000 INFO: Building COLLECT because out00-COLLECT.toc is non existent
6000 INFO: Building COLLECT out00-COLLECT.toc
8703 INFO: Building COLLECT out00-COLLECT.toc completed successfully.
Any help is greatly appreciated.
Currently PyInstaller 3.3.1 is not compatible with the latest version of PyQt5 (after version 5.9.2). They are working on a new PyInstaller hook for PyQt5.
If you are using Pyinstaller 3.3 or 3.3.1 then you have to use these versions of SIP and PyQt5 (order of installation matters).
On Windows:
sip-4.19.6-cp36-none-win32.whl
PyQt5-5.9.2-5.9.3-cp35.cp36.cp37-none-win32.whl
pyqt5_tools-5.9.0.1.2-cp36-none-win32.whl (if you need the PyQt designer)
On macOS:
sip-4.19.8-cp36-cp36m-macosx_10_6_intel.whl
PyQt5-5.9.2-5.9.3-cp35.cp36.cp37-abi3-macosx_10_6_intel.whl
I've written a Python program using phonenumbers. On my Windows system with Python 3.6.4 installed, it works as intended, but I'd like to redistribute it as a standalone Windows executable, so I've run PyInstaller on it to get a Windows executable.
However, I'm hitting two barriers: first, I'm a rank beginner with Python, and the details of the Python environment are still a slog for me.
Second, the .EXE program throws an exception trying to dynamically load a module (country data to parse a number, FWIW), and I haven't managed to figure out how to incorporate dynamically loaded modules into a standalone executable. (The failure was in a "try" clause, so it took me a while to locate it.) My own system has all the memory I'd need to pre-load all the needed dynamically loaded modules up front, so I'd prefer to do things that way.
Solutions, suggestions, hints, thoughts? A fully-worked example somewhere? Perhaps I'm failing to understand an answer already in the forum, but that would be because I'm a beginner, so I'd appreciate some clarity.
____
Thanks for your notes. To answer #adnan-y and #apogalacticon: I've made a simple test case. Here are the program source, output from running the script, the pyinstaller run on the program (with no prior build directories or files), its warning file, the result of running the executable, and the location of the file that the Python script correctly loads and correctly uses on the sample input.
The source program Parse-failure.py
import phonenumbers
Number = "+17034820623"
print("The unprocessed number is <%s>" % (Number))
Parsed_Number = phonenumbers.parse(Number)
print("The parsed number is <%s>" % (Parsed_Number))
Output from running Parse-failure.py
$ Parse-failure.py
The unprocessed number is <+17034820623>
The parsed number is <Country Code: 1 National Number: 7034820623>
Pyinstaller run
$ PyInstaller --onefile Parse-failure.py
94 INFO: PyInstaller: 3.4.dev0
94 INFO: Python: 3.6.4
94 INFO: Platform: Windows-7-6.1.7601-SP1
95 INFO: wrote N:\PPR\Parse-failure.spec
96 INFO: UPX is not available.
97 INFO: Extending PYTHONPATH with paths
['f:\\Dems\\PhoneCheck', 'f:\\Dems\\PhoneCheck']
98 INFO: checking Analysis
98 INFO: Building Analysis because out00-Analysis.toc is non existent
98 INFO: Initializing module dependency graph...
99 INFO: Initializing module graph hooks...
101 INFO: Analyzing base_library.zip ...
5155 INFO: running Analysis out00-Analysis.toc
5157 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by d:\python36\python.exe
6356 INFO: Caching module hooks...
6363 INFO: Analyzing N:\PPR\Parse-failure.py
6738 INFO: Loading module hooks...
6739 INFO: Loading module hook "hook-encodings.py"...
6843 INFO: Loading module hook "hook-pydoc.py"...
6845 INFO: Loading module hook "hook-xml.py"...
7238 INFO: Looking for ctypes DLLs
7238 INFO: Analyzing run-time hooks ...
7247 INFO: Looking for dynamic libraries
7372 INFO: Looking for eggs
7372 INFO: Using Python library d:\python36\python36.dll
7372 INFO: Found binding redirects:
[]
7375 INFO: Warnings written to N:\PPR\build\Parse-failure\warnParse-failure.txt
7461 INFO: Graph cross-reference written to N:\PPR\build\Parse-failure\xref-Parse-failure.html
7471 INFO: checking PYZ
7471 INFO: Building PYZ because out00-PYZ.toc is non existent
7471 INFO: Building PYZ (ZlibArchive) N:\PPR\build\Parse-failure\out00-PYZ.pyz
8292 INFO: Building PYZ (ZlibArchive) N:\PPR\build\Parse-failure\out00-PYZ.pyz completed successfully.
8302 INFO: checking PKG
8302 INFO: Building PKG because out00-PKG.toc is non existent
8302 INFO: Building PKG (CArchive) out00-PKG.pkg
10218 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
10222 INFO: Bootloader d:\python36\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
10222 INFO: checking EXE
10222 INFO: Building EXE because out00-EXE.toc is non existent
10222 INFO: Building EXE from out00-EXE.toc
10222 INFO: Appending archive to EXE N:\PPR\dist\Parse-failure.exe
10228 INFO: Building EXE from out00-EXE.toc completed successfully.
Warnings file build\PCS\warnParse-failure.txt
missing module named resource - imported by posix, n:\PPR\Parse-failure.py
missing module named posix - imported by os, n:\PPR\Parse-failure.py
missing module named _posixsubprocess - imported by subprocess, n:\PPR\Parse-failure.py
missing module named org - imported by pickle, n:\PPR\Parse-failure.py
missing module named readline - imported by cmd, code, pdb, n:\PPR\Parse-failure.py
excluded module named _frozen_importlib - imported by importlib, importlib.abc, n:\PPR\Parse-failure.py
missing module named _frozen_importlib_external - imported by importlib._bootstrap, importlib, importlib.abc, n:\PPR\Parse-failure.py
missing module named _winreg - imported by platform, n:\PPR\Parse-failure.py
missing module named _scproxy - imported by urllib.request
missing module named java - imported by platform, n:\PPR\Parse-failure.py
missing module named 'java.lang' - imported by platform, n:\PPR\Parse-failure.py, xml.sax._exceptions
missing module named vms_lib - imported by platform, n:\PPR\Parse-failure.py
missing module named termios - imported by tty, n:\PPR\Parse-failure.py, getpass
missing module named grp - imported by shutil, tarfile, n:\PPR\Parse-failure.py
missing module named pwd - imported by posixpath, shutil, tarfile, http.server, webbrowser, n:\PPR\Parse-failure.py, netrc, getpass
missing module named _dummy_threading - imported by dummy_threading, n:\PPR\Parse-failure.py
missing module named 'org.python' - imported by copy, n:\PPR\Parse-failure.py, xml.sax
Run the executable
$ dist\Parse-failure.exe
[1364] Failed to execute script Parse-failure
The unprocessed number is <+17034820623>
Traceback (most recent call last):
File "Parse-failure.py", line 7, in <module>
Parsed_Number = phonenumbers.parse(Number)
File "site-packages\phonenumbers\phonenumberutil.py", line 2835, in parse
File "site-packages\phonenumbers\phonemetadata.py", line 294, in metadata_for_region_or_calling_code
File "site-packages\phonenumbers\phonemetadata.py", line 267, in metadata_for_region
File "site-packages\phonenumbers\data\__init__.py", line 23, in _load_region
ModuleNotFoundError: No module named 'phonenumbers.data.region_US'
The location of ...region_US
D:\Python36\Lib\site-packages\phonenumbers\data\region_US.py
Hello, everyone.
I am trying to make an .exe package for a small desktop app.
I am using python 35, PyQt5 and Pyinstaller 3.4.dev0+97ce49bad.
When I run Pyinstaller, it shows this:
D:\SlowControl>pyinstaller --onefile --paths D:\Python35\Li
b\site-packages Slow_control.py
90 INFO: PyInstaller: 3.4.dev0+97ce49bad
90 INFO: Python: 3.5.2
90 INFO: Platform: Windows-7-6.1.7601-SP1
90 INFO: wrote D:\SlowControl\Slow_control.spec
90 INFO: UPX is not available.
115 INFO: Extending PYTHONPATH with paths
['D:\\SlowControl',
'D:\\Python35\\Lib\\site-packages',
'D:\\SlowControl']
115 INFO: checking Analysis
115 INFO: Building Analysis because out00-Analysis.toc is non existent
115 INFO: Initializing module dependency graph...
120 INFO: Initializing module graph hooks...
125 INFO: Analyzing base_library.zip ...
4295 INFO: running Analysis out00-Analysis.toc
5185 INFO: Caching module hooks...
5190 INFO: Analyzing D:\SlowControl\Slow_control.py
5305 INFO: Processing pre-find module path hook distutils
5460 INFO: Processing pre-safe import module hook six.moves
7845 INFO: Processing pre-find module path hook site
7845 INFO: site: retargeting to fake-dir 'D:\\Python35\\lib\\site-packages\\pyin
staller-3.4.dev0+97ce49bad-py3.5.egg\\PyInstaller\\fake-modules'
7880 INFO: Processing pre-safe import module hook win32com
15995 INFO: Loading module hooks...
15995 INFO: Loading module hook "hook-pkg_resources.py"...
16610 INFO: Loading module hook "hook-encodings.py"...
16780 INFO: Loading module hook "hook-numpy.core.py"...
16860 INFO: Loading module hook "hook-pytz.py"...
16930 INFO: Loading module hook "hook-pythoncom.py"...
17160 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
17695 INFO: Loading module hook "hook-xml.py"...
17695 INFO: Loading module hook "hook-sysconfig.py"...
17695 INFO: Loading module hook "hook-matplotlib.py"...
18140 INFO: Loading module hook "hook-distutils.py"...
18150 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
18235 INFO: Loading module hook "hook-lib2to3.py"...
18290 INFO: Loading module hook "hook-pywintypes.py"...
18535 INFO: Loading module hook "hook-PyQt5.py"...
18535 INFO: Loading module hook "hook-PyQt5.QtWidgets.py"...
18540 INFO: Loading module hook "hook-matplotlib.backends.py"...
19220 INFO: Matplotlib backend "GTK": ignored
Gtk* backend requires pygtk to be installed.
20280 INFO: Matplotlib backend "GTKAgg": ignored
Gtk* backend requires pygtk to be installed.
20690 INFO: Matplotlib backend "GTKCairo": ignored
No module named 'gtk'
21155 INFO: Matplotlib backend "MacOSX": ignored
cannot import name '_macosx'
21975 INFO: Matplotlib backend "Qt4Agg": added
22580 INFO: Matplotlib backend "Qt5Agg": added
23210 INFO: Matplotlib backend "TkAgg": added
23700 INFO: Matplotlib backend "WX": ignored
Matplotlib backend_wx and backend_wxagg require wxPython >=2.8.12
24165 INFO: Matplotlib backend "WXAgg": ignored
Matplotlib backend_wx and backend_wxagg require wxPython >=2.8.12
24490 INFO: Matplotlib backend "GTK3Cairo": ignored
Cairo backend requires that cairocffi or pycairo is installed.
24960 INFO: Matplotlib backend "GTK3Agg": ignored
Gtk3 backend requires pygobject to be installed.
25280 INFO: Matplotlib backend "WebAgg": ignored
Traceback (most recent call last):
File "D:\Python35\Lib\site-packages\matplotlib\backends\backend_webagg.py", li
ne 29, in <module>
import tornado
ImportError: No module named 'tornado'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 12, in <module>
File "D:\Python35\Lib\site-packages\matplotlib\backends\backend_webagg.py", li
ne 31, in <module>
raise RuntimeError("The WebAgg backend requires Tornado.")
RuntimeError: The WebAgg backend requires Tornado.
25625 INFO: Matplotlib backend "nbAgg": ignored
No module named 'tornado'
26095 INFO: Matplotlib backend "agg": added
26420 INFO: Matplotlib backend "cairo": ignored
Cairo backend requires that cairocffi or pycairo is installed.
26740 INFO: Matplotlib backend "gdk": ignored
No module named 'gobject'
27245 INFO: Matplotlib backend "pdf": added
27810 INFO: Matplotlib backend "pgf": added
28390 INFO: Matplotlib backend "ps": added
28870 INFO: Matplotlib backend "svg": added
29335 INFO: Matplotlib backend "template": added
29925 INFO: Loading module hook "hook-win32com.py"...
30030 INFO: Loading module hook "hook-setuptools.py"...
30030 WARNING: Hidden import "setuptools.msvc" not found!
30030 INFO: Loading module hook "hook-pydoc.py"...
30040 INFO: Loading module hook "hook-_tkinter.py"...
30220 INFO: checking Tree
30220 INFO: Building Tree because out00-Tree.toc is non existent
30220 INFO: Building Tree out00-Tree.toc
30275 INFO: checking Tree
30275 INFO: Building Tree because out01-Tree.toc is non existent
30275 INFO: Building Tree out01-Tree.toc
30280 INFO: Loading module hook "hook-PyQt5.Qt.py"...
30285 INFO: Loading module hook "hook-PyQt5.QtPrintSupport.py"...
30420 INFO: checking Tree
30420 INFO: Building Tree because out02-Tree.toc is non existent
30420 INFO: Building Tree out02-Tree.toc
30430 INFO: Looking for ctypes DLLs
30505 INFO: Analyzing run-time hooks ...
30515 INFO: Including run-time hook 'pyi_rth_qt5.py'
30530 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
30535 INFO: Including run-time hook 'pyi_rth_win32comgenpy.py'
30540 INFO: Including run-time hook 'pyi_rth_pkgres.py'
30540 INFO: Including run-time hook 'pyi_rth__tkinter.py'
30545 INFO: Including run-time hook 'pyi_rth_mplconfig.py'
30545 INFO: Including run-time hook 'pyi_rth_mpldata.py'
30565 INFO: Looking for dynamic libraries
35095 INFO: Looking for eggs
35095 INFO: Using Python library D:\Python35\python35.dll
35095 INFO: Found binding redirects:
[]
35130 INFO: Warnings written to D:\SlowControl\build\Slow_c
ontrol\warnSlow_control.txt
35360 INFO: Graph cross-reference written to D:\SlowControl
\build\Slow_control\xref-Slow_control.html
35460 INFO: checking PYZ
35460 INFO: Building PYZ because out00-PYZ.toc is non existent
35460 INFO: Building PYZ (ZlibArchive) D:\SlowControl\build
\Slow_control\out00-PYZ.pyz
37900 INFO: Building PYZ (ZlibArchive) D:\SlowControl\build
\Slow_control\out00-PYZ.pyz completed successfully.
37950 INFO: checking PKG
37955 INFO: Building PKG because out00-PKG.toc is non existent
37955 INFO: Building PKG (CArchive) out00-PKG.pkg
58380 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
58465 INFO: Bootloader D:\Python35\lib\site-packages\pyinstaller-3.4.dev0+97ce49
bad-py3.5.egg\PyInstaller\bootloader\Windows-64bit\run.exe
58465 INFO: checking EXE
58465 INFO: Building EXE because out00-EXE.toc is non existent
58465 INFO: Building EXE from out00-EXE.toc
58465 INFO: Appending archive to EXE D:\SlowControl\dist\Sl
ow_control.exe
58520 INFO: Building EXE from out00-EXE.toc completed successfully.
When I try to start SlowControl.exe it shows
Traceback (most recent call last):
File "Slow_control.py", line 2, in <module>
File "D:\Python35\lib\site-packages\pyinstaller-3.4.dev0+97ce49bad-py3.5.egg\P
yInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "Lib\site-packages\matplotlib\pyplot.py", line 109, in <module>
File "Lib\site-packages\matplotlib\pyplot.py", line 91, in _backend_selection
File "D:\Python35\lib\site-packages\pyinstaller-3.4.dev0+97ce49bad-py3.5.egg\P
yInstaller\loader\pyimod03_importers.py", line 714, in load_module
module = loader.load_module(fullname)
ImportError: DLL load failed: The specified procedure could not be found.
[1596] Failed to execute script Slow_control
I am not sure but maybe the problem is with hidden imports?
If so, could anyone please explain which files are hidden? I just don't understand.
Thanks in advance!
I'm trying to build the following script that uses pycryptodome:
# based on this example http://www.codekoala.com/posts/aes-encryption-python-using-pycrypto/#comment-25921785
from Crypto.Cipher import AES
from Crypto import Random
import base64
plain_text = 'Secret data'
block_size = 16
key_size = 32
mode = AES.MODE_CBC
key_bytes = Random.get_random_bytes(key_size)
pad = block_size - len(plain_text) % block_size
data = plain_text + pad * chr(pad)
iv_bytes = Random.get_random_bytes(block_size)
encrypted_bytes = iv_bytes + AES.new(key_bytes, mode, iv_bytes).encrypt(bytes(data, encoding='utf-8'))
encrypted_string = base64.urlsafe_b64encode(encrypted_bytes)
key_string = base64.urlsafe_b64encode(key_bytes)
key_bytes2 = base64.urlsafe_b64decode(key_string)
assert key_bytes == key_bytes2
encrypted_bytes2 = base64.urlsafe_b64decode(encrypted_string)
assert encrypted_bytes == encrypted_bytes2
iv_bytes2 = encrypted_bytes2[:block_size]
assert iv_bytes == iv_bytes2
encrypted_bytes2 = encrypted_bytes2[block_size:]
plain_text2 = AES.new(key_bytes2, mode, iv_bytes2).decrypt(encrypted_bytes2)
print(plain_text2)
pad = int(plain_text2[-1])
print(pad)
plain_text2 = plain_text2[:-pad].decode('utf-8')
print(plain_text2)
assert plain_text == plain_text2
This is the output I get while running pyinstaller:
C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec>pyinstaller crypto.py
46 INFO: PyInstaller: 3.3
46 INFO: Python: 3.6.2
46 INFO: Platform: Windows-7-6.1.7601-SP1
46 INFO: wrote C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\crypto.spec
46 INFO: UPX is not available.
62 INFO: Extending PYTHONPATH with paths
['C:\\Users\\test\\Documents\\MiniCryptoCodec\\minicryptocodec',
'C:\\Users\\test\\Documents\\MiniCryptoCodec\\minicryptocodec']
62 INFO: checking Analysis
62 INFO: Building Analysis because out00-Analysis.toc is non existent
62 INFO: Initializing module dependency graph...
62 INFO: Initializing module graph hooks...
62 INFO: Analyzing base_library.zip ...
2718 INFO: running Analysis out00-Analysis.toc
2718 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by c:\users\test\appdata\local\programs\python\python36\python.exe
3281 INFO: Caching module hooks...
3281 INFO: Analyzing C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\crypto.py
3625 INFO: Loading module hooks...
3625 INFO: Loading module hook "hook-encodings.py"...
3703 INFO: Loading module hook "hook-pydoc.py"...
3703 INFO: Loading module hook "hook-xml.py"...
3921 INFO: Looking for ctypes DLLs
3921 INFO: Analyzing run-time hooks ...
3921 INFO: Looking for dynamic libraries
4000 INFO: Looking for eggs
4000 INFO: Using Python library c:\users\test\appdata\local\programs\python\python36\python36.dll
4000 INFO: Found binding redirects:
[]
4015 INFO: Warnings written to C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\build\crypto\warncrypto.txt
4062 INFO: Graph cross-reference written to C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\build\crypto\xref-crypto.html
4062 INFO: checking PYZ
4062 INFO: Building PYZ because out00-PYZ.toc is non existent
4062 INFO: Building PYZ (ZlibArchive) C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\build\crypto\out00-PYZ.pyz
4515 INFO: Building PYZ (ZlibArchive) C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\build\crypto\out00-PYZ.pyz completed successfully.
4515 INFO: checking PKG
4515 INFO: Building PKG because out00-PKG.toc is non existent
4515 INFO: Building PKG (CArchive) out00-PKG.pkg
4531 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
4531 INFO: Bootloader c:\users\test\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
4531 INFO: checking EXE
4531 INFO: Building EXE because out00-EXE.toc is non existent
4531 INFO: Building EXE from out00-EXE.toc
4531 INFO: Appending archive to EXE C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec\build\crypto\crypto.exe
4546 INFO: Building EXE from out00-EXE.toc completed successfully.
4546 INFO: checking COLLECT
4546 INFO: Building COLLECT because out00-COLLECT.toc is non existent
4546 INFO: Building COLLECT out00-COLLECT.toc
4875 INFO: Building COLLECT out00-COLLECT.toc completed successfully.
Everything seems ok. But when I run the build executable I get this:
C:\Users\test\Documents\MiniCryptoCodec\minicryptocodec>dist\crypto\crypto.exe
Traceback (most recent call last):
File "crypto.py", line 1, in <module>
from Crypto.Cipher import AES
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "c:\users\test\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\Crypto\Cipher\__init__.py", line 3, in <module>
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "c:\users\test\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\Crypto\Cipher\_mode_ecb.py", line 46, in <module>
File "site-packages\Crypto\Util\_raw_api.py", line 189, in load_pycryptodome_raw_lib
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb'
[2824] Failed to execute script crypto
I even added 'pycryptodome' to the hiddenimports list in the spec file but it still does not work.
What do I need to do to build a working executable that uses pycryptodome?
You may need a more recent version of PyInstaller - there was a 'hook' added for pycryptodome three months ago
https://github.com/pyinstaller/pyinstaller/tree/develop/PyInstaller/hooks
After adding the hook I had additional issues and was able to get it working by installing cffi package
https://pypi.python.org/pypi/cffi
pip install cffi
If you want to solve your problems use pycryptodomex
pip uninstall -y pycryptodome
pip install pycryptodomex
Then search and all the imports of Crypto and replace them with Cryptodome.
It works using pyinstaller 3.4 as installed by pip on conda 4.5.12:
>conda create -n pyinstaller python=3.7
>activate pyinstaller
>pip install pyinstaller
>pip install pycryptodome
>conda list
# packages in environment at C:\Users\pablo\AppData\Local\Continuum\anaconda3\envs\pyinstaller:
#
# Name Version Build Channel
altgraph 0.16.1 <pip>
ca-certificates 2018.03.07 0
certifi 2018.11.29 py37_0
future 0.17.1 <pip>
macholib 1.11 <pip>
openssl 1.1.1a he774522_0
pefile 2018.8.8 <pip>
pip 18.1 py37_0
pycryptodome 3.7.2 <pip>
PyInstaller 3.4 <pip>
python 3.7.2 h8c8aaf0_0
pywin32-ctypes 0.2.0 <pip>
setuptools 40.6.3 py37_0
sqlite 3.26.0 he774522_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_0
wheel 0.32.3 py37_0
wincertstore 0.2 py37_0
>
>pyinstaller main.py
62 INFO: PyInstaller: 3.4
62 INFO: Python: 3.7.2
62 INFO: Platform: Windows-10-10.0.15063-SP0
62 INFO: wrote C:\Users\pablo\Dev\untitled\main.spec
78 INFO: UPX is not available.
78 INFO: Extending PYTHONPATH with paths
['C:\\Users\\pablo\\Dev\\untitled', 'C:\\Users\\pablo\\Dev\\untitled']
78 INFO: checking Analysis
78 INFO: Building Analysis because Analysis-00.toc is non existent
78 INFO: Initializing module dependency graph...
78 INFO: Initializing module graph hooks...
78 INFO: Analyzing base_library.zip ...
2734 INFO: running Analysis Analysis-00.toc
2734 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by c:\users\pablo\appdata\local\continuum\anaconda3\envs\pyinstaller\python.exe
3390 INFO: Caching module hooks...
3406 INFO: Analyzing C:\Users\pablo\Dev\untitled\main.py
3671 INFO: Loading module hooks...
3671 INFO: Loading module hook "hook-Crypto.py"...
3687 INFO: Loading module hook "hook-encodings.py"...
3765 INFO: Loading module hook "hook-pydoc.py"...
3765 INFO: Loading module hook "hook-xml.py"...
3953 INFO: Looking for ctypes DLLs
3953 INFO: Analyzing run-time hooks ...
3968 INFO: Looking for dynamic libraries
4859 INFO: Looking for eggs
4859 INFO: Using Python library c:\users\pablo\appdata\local\continuum\anaconda3\envs\pyinstaller\python37.dll
4859 INFO: Found binding redirects:
[]
4875 INFO: Warnings written to C:\Users\pablo\Dev\untitled\build\main\warn-main.txt
4906 INFO: Graph cross-reference written to C:\Users\pablo\Dev\untitled\build\main\xref-main.html
4937 INFO: checking PYZ
4937 INFO: Building PYZ because PYZ-00.toc is non existent
4937 INFO: Building PYZ (ZlibArchive) C:\Users\pablo\Dev\untitled\build\main\PYZ-00.pyz
5484 INFO: Building PYZ (ZlibArchive) C:\Users\pablo\Dev\untitled\build\main\PYZ-00.pyz completed successfully.
5499 INFO: checking PKG
5499 INFO: Building PKG because PKG-00.toc is non existent
5499 INFO: Building PKG (CArchive) PKG-00.pkg
5515 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
5515 INFO: Bootloader c:\users\pablo\appdata\local\continuum\anaconda3\envs\pyinstaller\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
5515 INFO: checking EXE
5515 INFO: Building EXE because EXE-00.toc is non existent
5515 INFO: Building EXE from EXE-00.toc
5515 INFO: Appending archive to EXE C:\Users\pablo\Dev\untitled\build\main\main.exe
5515 INFO: Building EXE from EXE-00.toc completed successfully.
5531 INFO: checking COLLECT
5531 INFO: Building COLLECT because COLLECT-00.toc is non existent
5531 INFO: Building COLLECT COLLECT-00.toc
6062 INFO: Building COLLECT COLLECT-00.toc completed successfully.
my environment:
python 3.6
pyinstaller 4.10
pycryptodomex 3.15.0
and then copy the hook file(hook-Cryptodome.py) into venv/Lib/site-packages/PyInstaller. then using the pyinstaller to excute following bash:
pyinstaller ../../demo.py -F -p ../venv/Lib/site-packages
ps. the path maybe you should adjust in your own PC.