Python pkg_resources and file access in packages - python-3.x

I'm building my first python package (which I then install with pip) and I need to use some non-python files. In these answers, it is explained that I should use the pkg_resources function. But I can't figure out a working example. Let say I have this project structure:
package_name/
----data/
--------image.png
----package_name/
--------__init__.py
--------file.py
----setup.py
----MANIFEST.in
----conf.yml
Now I want to access conf.yml and image.png from file.py. How should I proceed in:
file.py ?
setup.py ?
MANIFEST.in ?

The simplest way to access these files would be to include in MANIFEST.in
global-include *.png
global-include *.yml
MANIFEST.in will only use files for a source distribution though, while setup.py will include files for binary/wheel distributions so just to be safe, inside of your setup.py
include_package_data = True,
package_data = {
'' : ['*.png'],
'' : ['*.yml'],
}
Then you can reference the specific file like so from file.py
from pkg_resources import resource_string
def foo():
pngfile = resource_string(__name__, 'data/image.png')
ymlfile = resource_string(__name__, 'conf.yml')
Notice how for the png file I've specified the directory.
This solution also does not account for files of the same extension which you may want to exclude, but those could easily be taken care of with exclude or specifying filenames rather than using the asterisk.
I know there are questions that could easily be considered duplicates, but I had trouble getting a workable example as well and after a good while badgering away myself I managed to get something to work, and this was it.

Related

Souce code getting packaged in python wheel

We are using the wheels to deploy our code to QA/Production. Recently we found/realized that wheel packages are actually storing our source code. And by simple command as below will open all the source code inside it.
unzip package.whl
command used for wheel creation is as below
cd /path/to/source/code/folder
python setup.py bdist bdist_wheel
So,
Is there any way to create wheels which creates binary and stores in package rather than source code?
In the simplest sense, wheel is just:
a zip file
with a specific filename
and a specific directory layout
containing pure-Python source code
and any platform-specific binaries
This means that a wheel (and any other distribution) is not a binary itself, but it may contain platform-specific binaries -- for example, if you are building/compiling some C code along with your Python package.
Most wheels are pure-Python, which means that they only contain Python source code.
It seems like you're asking how to "compile" Python code into an obfuscated binary. This is not the goal of a wheel. You might want to read more details on the wheel format here: https://www.python.org/dev/peps/pep-0427/
Is there any way to create wheels which creates binary and stores in package rather than source code?
Not with the wheel format. If this is actually your goal, you may want to look into pyinstaller, py2exe or cython, depending on the target platform.
In a case someone stumbles here the same way I did. If you
Use Cython to pre-compile your library.
Have not only .pyx, but also .py modules. For example you want to do this with some existing project without any modifications (except for setup.py), or consider it unreasonable to do in the first place as Cython consumes .py files also).
Want to distribute pre-compiled library without any .py files included (except for may be empty __init__.py files).
Then, you can apply the following (quite dirty) solution to exclude any files you want from the wheel:
from wheel.bdist_wheel import bdist_wheel
class CommandBdistWheel(bdist_wheel):
# Called almost exactly before filling `.whl` archive
def write_wheelfile(self, *args, **kwargs):
dr = f'{self.bdist_dir}/<package name>'
paths = [
path for path in glob.glob(f'{dr}/**/*.py', recursive=True)
if os.path.basename(path) != '__init__.py'
]
for path in paths:
os.remove(path)
super().write_wheelfile(*args, **kwargs)
setup(
# ...
cmdclass={'bdist_wheel': CommandBdistWheel},
# ...
)

How to distribute python based software on Linux based OS

condensed version of what I want to achieve:
Create .rpm and .deb packages from my source.py source code and make sure all dependencies get resolved when installing them on an deb/rpm based linux distribution.
More details:
Let's assume I have created a piece of software which is located in a folder structure like this:
---MyProgram Folder
---MyProgram Folder
---img Folder
---logo.ico File
---media Folder
---head.txt File
---__init__.py File
---source.py File
---a.py File
---LICENSE File
---README.md File
---setup.py File
The file setup.py contains the following:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="MyProgram",
version="0.0.1",
author="First Last",
author_email="email#memore.com",
description="A tool to create nice things",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://google.com",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.7',
data_files=[
('.../MyProgram/img/logo.ico'),
('.../MyProgram/media/head.txt'),
],
)
I now run
python setup.py sdist bdist_rpm
from a cmd line under '.../MyProgram'. Two folders 'dist' and 'build' are created as well as 'MyProgram.tar.gz' and two rpm's 'MyProgram-noarch.rpm' and 'MyProgram-src.rpm'.
When i try to install 'noarch.rpm' under fedora 31 the process end successfully but no "shortcut" is created, and when i type MyProgram in a cmd line it is not found.
rpm -ql MyFilter
does find it and outputs a bunch of paths:
/usr/lib/python3.7/site-packages/MyProgram/...
/usr/lib/python3.7/site-packages/MyProgram/source.py
/usr/lib/python3.7/site-packages/MyProgram/a.py
....
Which tells me that my installation at least has copied the basic filesystem. But i also see that all the original .py files are still .py files.
My questions:
How can i 'make' the rpm so that all dependencies are contained inside the rpm, or at least get resolved by dnf/apt/yum when installing the rpm? In other wording: Is it possible to bundle all dependencies into a rpm/deb like in an .exe for example?
How can i specify a path like '/usr/bin' or 'usr/share' as installation target
dir?
How can i add a launcher app bundled into the rpm/deb?
Is the above a good way of doing this at all?
If the solution to this is trivial and i just overlooked it i am really sorry to bother you but atm i just can't see it. Sites that have relevant information and that i already reviewed:
https://docs.python.org/2.0/dist/creating-rpms.html
https://github.com/AppImage/AppImageKit/wiki/Bundling-Python-apps
Python 3.5 create .rpm with pyinstaller generated executable
https://github.com/junaruga/rpm-py-installer
https://www.pyinstaller.org/
https://packaging.python.org/overview/#python-source-distributions
https://packaging.python.org/overview/
https://pyinstaller.readthedocs.io/en/stable/usage.html
https://pyinstaller.readthedocs.io/en/stable/installation.html
https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html
Just my two cents, rather than a complete answer. Will mostly touch on RPM packaging.
The bdist_rpm option seems easy, but you have little control of the logic of the .spec file it generates/uses and cannot do fancy stuff like scriplets, etc.
That is, unless you take the approach of having it generate the .spec file and quit (instead of building final RPM). From the docs:
If you wish, you can separate these three steps. You can use the --spec-only option to make bdist_rpm just create the .spec file and exit; in this case, the .spec file will be written to the “distribution directory”—normally dist/, but customizable with the --dist-dir option. (Normally, the .spec file winds up deep in the “build tree,” in a temporary directory created by bdist_rpm.)
But as a matter of preference and consistency, I would advise on following distro-specific guidelines for packaging Python apps.
In that way, you will be more in line with the distro's you are building for.
It is not the easiest way though. You will have to shift through some docs. Basically, if you're building for anything CentOS/RHEL, Fedora guidelines for packaging should be observed.
You can find the extra reference here, with the example .spec file for building both Python 2 and 3 versions of the same app.
For this whole 'build like a distro' thing, you would definitely want to look into using mock for the job, to build your package in a chroot.
As for the "shortcut" issue, you have to have your setup.py declare some console scripts for it to create one when you install your package. E.g. from lastversion's setup.py:
entry_points={"console_scripts": ["lastversion = lastversion:main"]},
This entry will result in a "binary" lastversion created/installed (which runs the defined function) when you install your Python package.
Subsequently, in the spec files, the macro %py2_install will make use of setup.py to create the same launcher program.
And you will then be able to ensure that launcher is packaged by placing it in the files section of the spec file:
%files -n python3-myapp
%license COPYING
%doc README.rst
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}-*.egg-info/
%{_bindir}/myapp

Copy non python files via package_data to Scripts directory

I have some scripts in my package, that rely on some template xml files.
Those scripts are callable by entry points and I wanted to reference the template files by a relative path.
When calling the script via python -m ... the scripts themselves are called from within lib\site-packages and there the xml files are available as I put them in my setup.py like this:
setup(
...
packages=['my_pck'],
package_dir={'my_pck': 'python/src/my_pck'},
package_data={'my_pck': ['reports/templates/*.xml']},
...
)
I know, I could copy those templates also by using data_files in my setup.py but using package_data seems better to me.
Unfortunately package_data seems not to copy those files to the Scripts folder where the entry points are located.
So my question is, is this even achievable via package_data and if, how?
Or is there a more pythonic, easier way to achieve this? Maybe not referencing those files via paths relative to the scripts?
Looks like importlib-resources might help here. This library is able to find the actual path to a resource file packaged as package_data by setuptools.
Access the package_data files from your code with something like this:
with importlib_resources.path('my_pck.reports.templates', 'a.xml') as xml_path:
do_something(xml_path)

Is it possible to have a setup.py instead of a SConstruct?

I am taking a close look at Scons and something smells. SCons uses SConstruct files as base configuration file. This configuration file is a Python file but:
It does not have the .py extension
It does not have any import directives
It is not possible to have auto-completion from IDEs
It it possible to use a variant of the SConstruct file where I could find something like the following?
# build.py
import scons
env = scons.Environment()
env.Program('foo')
It would be not simple (but possible) to do what you're asking. SConscripts are plain python, however the globals available in the context of SConstruct or SConscripts are carefully constructed.
Any user can add methods and also pass python objects into the SConscripts via Export() or exports (in a SConscript call).
That said try:
from SCons.Script import *
That should get you some of what you're looking for.
The fact that Pycharm cannot find the symbols in question doesn't mean the it is not a plain python file.
Additionally I'm not sure how the subject of your question relates to the contents of your question. Typically setup.py is a file to build packages ala setuptools and install via pip (or similar).
Sure you can build whatever you are trying to build with setuptools, it will likely be harder to do, but that said if you get it to work, perhaps easier to upload and distribute via pypi
p.s. It's SCons not Scons.

Packaging Multiple Python Files

I currently am using this guide to package up my project wasp. However currently everything lives inside of the wasp file.
That's not ideal. I would rather have all the classes in separate files so it can be more effectively managed. I have the series of files needed in the debian directory. But I'm not sure how to configure the packaging to package multiple files.
Is there a way to change my packaging to package more than just the one script file?
I'm not a debian package or Python expert, but one way would be to copy the various source files to another location (outside of /usr/bin), and then have /usr/bin/wasp call out to them.
Say you put all of your python code in src/ in the root of your repo. In the debian/install file, you'd have:
wasp usr/bin
src/* usr/lib/wasp/
You'd then just need /usr/bin/wasp to call some entry point in src. For example,
#!/usr/bin/python3
import sys
sys.path.append('/usr/lib/wasp/')
import wasp # or whatever you expose in src
# ...
Again, I don't know the best practices here (either in directory or python usage) but I think this would at least work!

Resources