pyinstaller dynamic module additions - python-3.x

I have a python project, that can dynamically load other modules based on user inputs by main. I have pyintaller building the exe fine along with all the required modules needed and this all works fine. But I have a directory filled with algorithms, built written too in python, that can be dynamically loaded based on user action from the main exe. How can OR can these files also be converted to binaries from the spec file.
I have a number of algorithms in 'share/smartsheetAlgo' that too need to be converted to binary files. Is there are way from the spec sheet, now they are included as py files
Here is my spec file
-- mode: python ; coding: utf-8 --
block_cipher = None
a = Analysis(['stecsmartsheet.py'],
pathex=['/home/ed/smartsheet','/usr/local/lib/python3.8'],
binaries=[],
datas=[('./share/original_filenames.txt','share'),
('./share/smartsheetAlgo/*.py','share/smartsheetAlgo'),
('./share/projects/*.prj','share/projects'),
('./share/icons/*.*','share/icons'),
('./share/icons/actions/*.*','share/icons/actions'),
('./share/icons/charts/*.*','share/icons/charts'),
('./share/icons/status/*.*','share/icons/status'),
('./share/libs/win32/*.*','.'),
],
hiddenimports=['VremAlgoinfo','parseline','statistics','pyqtgraph'],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='smartsheet',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True,
disable_windowed_traceback=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='smartsheet')

Related

Include sys.stderr in a Pyinstaller project

One of the subdependencies of my project is transformers. This only happened when I upgraded transformers from 4.16.0 to the latest version 4.25.1. When I try to compile the project with pyinstaller I get the following error:
Traceback (most recent call last):
File "main.py", line 14, in <module>
...
File "transformers\utils\import_utils.py", line 36, in <module>
File "transformers\utils\logging.py", line 123, in get_logger
File "transformers\utils\logging.py", line 86, in _configure_library_root_logger
AttributeError: 'NoneType' object has no attribute 'flush'
Upon further inspection I found the following function in logging.py. It seems that sys.stderr is being set as NoneType for some reason.
def _configure_library_root_logger() -> None:
global _default_handler
with _lock:
if _default_handler:
_default_handler = logging.StreamHandler()
_default_handler.flush = sys.stderr.flush # Error on this line
...
This is the file I'm using to compile the project:
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.utils.hooks import copy_metadata
datas = []
datas += copy_metadata('tqdm')
datas += copy_metadata('numpy')
a = Analysis(['.main.py'],
pathex=['.'],
binaries=[],
datas=datas,
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=None,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='MyApp',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=False,
disable_windowed_traceback=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
icon="icon.ico")
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='main')
I have tried setting the paths paramater: pathex=['.', 'path/to/env/Lib/site-packages']. I also tried including it as a hidden import: hiddenimports=['sys', 'sys.stderr']. But none of these seem to work. I know I can just downgrade, but I want to use the latest version.

Every time I run exe the new file getting created?

Hi I have converted my python code to exe using pyinstaller,
I'm converting my python code to single file using the below .spec file
# -*- mode: python ; coding: utf-8 -*-
block_cipher = None
a = Analysis(['obiee_pbi_connector\\migration.py'],
pathex=['D:\\Automation projects\\bimigration_demo_for_ms\\back_end_python'],
binaries=[],
datas=[('logging.ini','../.'),('pbi','../pbi'),('obiee_repo','../obiee_repo'),('config','../config'),('exe_dependencies/Lib/site-packages/tableauhyperapi','./tableauhyperapi'),('exe_dependencies/Lib/site-packages/tableauhyperapi-0.0.13394.dist-info','tableauhyperapi-0.0.13394.dist-info'),('exe_dependencies/Lib/site-packages/pythonjsonlogger','pythonjsonlogger'),('exe_dependencies/Lib/site-packages/python_json_logger-2.0.2.dist-info','python_json_logger-2.0.2.dist-info')
],
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='revive',
debug=True,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True,
disable_windowed_traceback=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None )
when you convert it into only one file the configuration and temporary files that we write/read will be present at the temp folder of current user
Where I'm using add data to create move my sqlite db, configuration files, and two libraries(because it's not getting packaged so I'm doing like this), whenever I run exe all these files getting copied to temp folder, so each time I run my application it's seems to be newly installed one, how can I resolve this.
Thanks in advance

What is the correct pyinstaller Command to make pysnmp executable work on windows

I am using the following:
pysnmp 4.4.9
Python 3.7.2
pyinstaller 3.4
My code is very simple, the heart of it is below:
def snmpv2Get(ip_address):
errorIndication, errorStatus, errorIndex, varBinds = next(
getCmd(SnmpEngine(),
CommunityData(snmpv2Community),
UdpTransportTarget([ip_address, 161]),
ContextData(),
ObjectType(ObjectIdentity('1.3.6.1.2.1.1.1.0')),
ObjectType(ObjectIdentity('1.3.6.1.2.1.1.2.0')),
lookupMib=False)
)
When converting the code to an executable using pyinstaller, I get this error trying to run the generated executable:
pysnmp.smi.error.MibNotFoundError: No module __SNMP-FRAMEWORK-MIB loaded at <pysnmp.smi.builder.MibBuilder object at 0x000002788101EA20>
The command I am using for pyinstaller is as follows:
pyinstaller -y -F -i "icon.ico" SNMP.py --hidden-import="pysnmp.smi.mibs,pysnmp.smi.mibs.instances,pysnmp.smi.exval,pysnmp.cache"
Honestly, I don't want to modify *.spec file because each time I modified it, the command overrides *.spec contents, I don't know why.
Kindly, what is the correct command line to avoid this "No module" error. I have seen many threads with similar error message, but all of them dealt with the *.spec file, what I am looking for is the correct command line to use on windows, not *.spec file.
It seems that PyInstaller could not resolve pysnmp by its own so an easy solution is to use Tree class and embedding the library directory within the executable. After generating spec file add the Tree class. So your spec file should look like something like this (remember to replace the module path according to your Python path):
# -*- mode: python -*-
block_cipher = None
a = Analysis(
...
)
a.datas += Tree("./env/Lib/site-packages/pysnmp", prefix='pysnmp')
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
...
And finally generate your executable with (I suggest you not use your script name same as module name):
pyinstaller script.spec
Here are the contents of my *.spec file, it works very well for pysnmp and pyinstaller:
# -*- mode: python -*-
# source: http://qaru.site/questions/8036799/pyinstaller-does-not-work-when-including-pysnmp
from PyInstaller.utils.hooks import collect_data_files, collect_submodules
x = Tree('C:/Python37/Lib/site-packages/pysnmp/smi/mibs',prefix='pysnmp/smi/mibs',excludes='.py')
block_cipher = None
a = Analysis(['SNMP.py'],
pathex=['path to python file/'],
binaries=[],
datas=[],
hiddenimports=['pysnmp.smi.exval','pysnmp.cache'] + collect_submodules('pysnmp.smi.mibs') + collect_submodules('pysnmp.smi.mibs.instances'),
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
x,
name='SNMP',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True , icon='icon.ico')

Pyinstaller on windows 10 cant find sub packages/modules

My project is structured as follows and I am having issues in using pyinstaller on windows 10 to build.
I am using python 3.7.3 and pyinstaller 3.4. PyQt5 version is 5.12.1
Pyinstaller builds properly with following command:
pyinstaller --clean -F calcrun.spec
However, it fails when I run calcrun.exe in dist/
#calcrun.py
import sys
from CalcApp import runner
from PyQt5 import QtWidgets
if __name__ == '__main__':
app = QtWidgets.QApplication(sys.argv)
application = runner.MyWindow()
sys.exit(app.exec())
Below is the error I get where it seems like its not able to see any module or sub package
C:\Users\Kiki\projects\learn_app\dist\calcrun>calcrun
Traceback (most recent call last):
File "learn_app\calcrun.py", line 2, in <module>
ModuleNotFoundError: No module named 'CalcApp'
[13160] Failed to execute script calcrun
I am also attaching here my calcrun.spec
# -*- mode: python -*-
block_cipher = None
a = Analysis(["calcrun.py"],
pathex=["C:\\Users\\Kiki\\projects\\learn_app"],
binaries=[],
datas=[],
hiddenimports=['pyqt5', 'pyqt5-sip'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='calcrun',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
name='calcrun')

ModuleNotFoundError for folder when using PyInstaller on Windows command line

I am building a kivy app on PyCharm. My Test.py imports testone.py and runs a function from it. When I run Test.py in PyCharm the app builds and runs fine. But then, to build the .exe for the app, I run the following in the command line:
python -m PyInstaller --name TestAppWD9 --onefile --add-data "Desktop\KivyUdemyCourse\AdvKivywKV\testone.py;Desktop\KivyUdemyCourse\AdvKivywKV\testone.py" C:\Users\jgldm\Desktop\KivyUdemyCourse\AdvKivywKV\Test.py
I get something like this as a spec file:
# -*- mode: python -*-
from kivy.deps import sdl2, glew
import pygame.pkgdata
block_cipher = None
a = Analysis(['C:\\Users\\jgldm\\Desktop\\KivyUdemyCourse\\AdvKivywKV\\Test.py'],
pathex=['C:\\Users\\jgldm'],
binaries=[],
datas=[('Desktop\\KivyUdemyCourse\\AdvKivywKV\\testone.py', 'Desktop\\KivyUdemyCourse\\AdvKivywKV\\testone.py')],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
*[Tree(p) for p in (sdl2.dep_bins + glew.dep_bins)],
[],
name='TestAppWD9',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
runtime_tmpdir=None,
console=True )
when I run the app built from the spec file I get ModuleNotFoundError: No module named 'AdvKivywKV'
How can I get the spec file to find and include AdvKivywKV?
The folder structure needed to be fixed by placing testone.py into a subfolder with an __init__.py file. The latter can be blank. After this I changed the import to from subfolder_name.testone import function

Resources