How to distribute python based software on Linux based OS - python-3.x

condensed version of what I want to achieve:
Create .rpm and .deb packages from my source.py source code and make sure all dependencies get resolved when installing them on an deb/rpm based linux distribution.
More details:
Let's assume I have created a piece of software which is located in a folder structure like this:
---MyProgram Folder
---MyProgram Folder
---img Folder
---logo.ico File
---media Folder
---head.txt File
---__init__.py File
---source.py File
---a.py File
---LICENSE File
---README.md File
---setup.py File
The file setup.py contains the following:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="MyProgram",
version="0.0.1",
author="First Last",
author_email="email#memore.com",
description="A tool to create nice things",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://google.com",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.7',
data_files=[
('.../MyProgram/img/logo.ico'),
('.../MyProgram/media/head.txt'),
],
)
I now run
python setup.py sdist bdist_rpm
from a cmd line under '.../MyProgram'. Two folders 'dist' and 'build' are created as well as 'MyProgram.tar.gz' and two rpm's 'MyProgram-noarch.rpm' and 'MyProgram-src.rpm'.
When i try to install 'noarch.rpm' under fedora 31 the process end successfully but no "shortcut" is created, and when i type MyProgram in a cmd line it is not found.
rpm -ql MyFilter
does find it and outputs a bunch of paths:
/usr/lib/python3.7/site-packages/MyProgram/...
/usr/lib/python3.7/site-packages/MyProgram/source.py
/usr/lib/python3.7/site-packages/MyProgram/a.py
....
Which tells me that my installation at least has copied the basic filesystem. But i also see that all the original .py files are still .py files.
My questions:
How can i 'make' the rpm so that all dependencies are contained inside the rpm, or at least get resolved by dnf/apt/yum when installing the rpm? In other wording: Is it possible to bundle all dependencies into a rpm/deb like in an .exe for example?
How can i specify a path like '/usr/bin' or 'usr/share' as installation target
dir?
How can i add a launcher app bundled into the rpm/deb?
Is the above a good way of doing this at all?
If the solution to this is trivial and i just overlooked it i am really sorry to bother you but atm i just can't see it. Sites that have relevant information and that i already reviewed:
https://docs.python.org/2.0/dist/creating-rpms.html
https://github.com/AppImage/AppImageKit/wiki/Bundling-Python-apps
Python 3.5 create .rpm with pyinstaller generated executable
https://github.com/junaruga/rpm-py-installer
https://www.pyinstaller.org/
https://packaging.python.org/overview/#python-source-distributions
https://packaging.python.org/overview/
https://pyinstaller.readthedocs.io/en/stable/usage.html
https://pyinstaller.readthedocs.io/en/stable/installation.html
https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html

Just my two cents, rather than a complete answer. Will mostly touch on RPM packaging.
The bdist_rpm option seems easy, but you have little control of the logic of the .spec file it generates/uses and cannot do fancy stuff like scriplets, etc.
That is, unless you take the approach of having it generate the .spec file and quit (instead of building final RPM). From the docs:
If you wish, you can separate these three steps. You can use the --spec-only option to make bdist_rpm just create the .spec file and exit; in this case, the .spec file will be written to the “distribution directory”—normally dist/, but customizable with the --dist-dir option. (Normally, the .spec file winds up deep in the “build tree,” in a temporary directory created by bdist_rpm.)
But as a matter of preference and consistency, I would advise on following distro-specific guidelines for packaging Python apps.
In that way, you will be more in line with the distro's you are building for.
It is not the easiest way though. You will have to shift through some docs. Basically, if you're building for anything CentOS/RHEL, Fedora guidelines for packaging should be observed.
You can find the extra reference here, with the example .spec file for building both Python 2 and 3 versions of the same app.
For this whole 'build like a distro' thing, you would definitely want to look into using mock for the job, to build your package in a chroot.
As for the "shortcut" issue, you have to have your setup.py declare some console scripts for it to create one when you install your package. E.g. from lastversion's setup.py:
entry_points={"console_scripts": ["lastversion = lastversion:main"]},
This entry will result in a "binary" lastversion created/installed (which runs the defined function) when you install your Python package.
Subsequently, in the spec files, the macro %py2_install will make use of setup.py to create the same launcher program.
And you will then be able to ensure that launcher is packaged by placing it in the files section of the spec file:
%files -n python3-myapp
%license COPYING
%doc README.rst
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}-*.egg-info/
%{_bindir}/myapp

Related

Souce code getting packaged in python wheel

We are using the wheels to deploy our code to QA/Production. Recently we found/realized that wheel packages are actually storing our source code. And by simple command as below will open all the source code inside it.
unzip package.whl
command used for wheel creation is as below
cd /path/to/source/code/folder
python setup.py bdist bdist_wheel
So,
Is there any way to create wheels which creates binary and stores in package rather than source code?
In the simplest sense, wheel is just:
a zip file
with a specific filename
and a specific directory layout
containing pure-Python source code
and any platform-specific binaries
This means that a wheel (and any other distribution) is not a binary itself, but it may contain platform-specific binaries -- for example, if you are building/compiling some C code along with your Python package.
Most wheels are pure-Python, which means that they only contain Python source code.
It seems like you're asking how to "compile" Python code into an obfuscated binary. This is not the goal of a wheel. You might want to read more details on the wheel format here: https://www.python.org/dev/peps/pep-0427/
Is there any way to create wheels which creates binary and stores in package rather than source code?
Not with the wheel format. If this is actually your goal, you may want to look into pyinstaller, py2exe or cython, depending on the target platform.
In a case someone stumbles here the same way I did. If you
Use Cython to pre-compile your library.
Have not only .pyx, but also .py modules. For example you want to do this with some existing project without any modifications (except for setup.py), or consider it unreasonable to do in the first place as Cython consumes .py files also).
Want to distribute pre-compiled library without any .py files included (except for may be empty __init__.py files).
Then, you can apply the following (quite dirty) solution to exclude any files you want from the wheel:
from wheel.bdist_wheel import bdist_wheel
class CommandBdistWheel(bdist_wheel):
# Called almost exactly before filling `.whl` archive
def write_wheelfile(self, *args, **kwargs):
dr = f'{self.bdist_dir}/<package name>'
paths = [
path for path in glob.glob(f'{dr}/**/*.py', recursive=True)
if os.path.basename(path) != '__init__.py'
]
for path in paths:
os.remove(path)
super().write_wheelfile(*args, **kwargs)
setup(
# ...
cmdclass={'bdist_wheel': CommandBdistWheel},
# ...
)

autoconf - how to generate template and copy it to proper directory

I'm using autoconf & automake in a C project. I'd like to create a .deb package, so I have the following control.in file:
Source: myproject
Section: misc
Priority: optional
Maintainer: Paul Walker <pwalk#test.it>
Build-Depends: debhelper (>=9), autotools-dev#MORE_DEPENDENCIES#
Standards-Version: 1.0.0
Homepage: https://www.my-website.it/
...
I'd like to configure this file with autoconf since there might be MORE_DEPENDENCIES according to some configure-time flag I did set.
This control.in file sits in a stubs/ folder, after substituting the MORE_DEPENDENCIES variable, I'd also like to copy the resulting control file into the final destination folder debian/ to create the .deb package.
So essentially I'm trying to have autoconf do the following:
Include {srcdir}/stubs/ as input folder in order to substitute macros in the .in file sitting there
Configure the stub {srcdir}/stubs/control.in, substitute whatever macro is in there, generate the file {srcdir}/stubs/control with substituted macros
Copy {srcdir}/stubs/control into the final destination folder {srcdir}/debian/control
I've been looking for examples or in the official documentation but still can't find how to configure a .in file outside of the root source folder.
Regarding 'how to copy the final file to the debian/ folder' I suppose I could use a symlink ? Is there any better way?
This is a top-of-the-foodchain problem.
The Debian packaging tools are designed to be the top-level tool for building Debian packages, and the control file is supposed to stay unchanged for the entire run of a package build. You can probably fudge your way around that if you only wish to build binary packages, but then there would be no point in modifying the build dependencies because they are evaluated earlier.
The normal build process will distclean the build tree in order to record changes in the source package, then rerun configure and build. If you update debian/control at this point, the results are undefined.
If you have to update debian/control programmatically, use a separate mechanism that is not reachable from the regular package build. If you have regular releases rather than cutting packages from development versions, it might even make more sense to treat the packaging as separate.

steps needed to create binary package for distribution in linux

I am little confused on how to create a complete binary package using rpmbuild from a project I just created (already compiled binary).
my current project contain similar format as this user (Packaging proprietary software for Linux)
Where I have
foo (binary)
data
libs
foo.sh
libs will contain all the shared libraries the project requires, and foo.sh is a script that sets LD_LIBRARY_PATH to include libs. Therefore, the user will execute foo.sh and the program should start.
I am looking at the tutorial from this site (rpm tutorial)
I understand to create a rpm I create a build area use rpmdev-setuptree
I can create a spec file use cd ~/rpmbuild/SPECS; rpmdev-newspec foo and if I got a good SOURCES folder I can build it with rpmbuild -ba foo.spec
But I have no idea how to setup the SOURCES directory. The tutorial stated (here) that I should create a tarball and place all my source file in it and put in SOURCE directory. What would be the source file in my case?
You are trying to create a RPM from binary files you have already? In that case, you can just leave the whole building stuff out of the SPEC file, and you need a SOURCE directory to keep the bundles you've got, the %prep step described below will take them from here.
In a binary package I built a while back from zip files, I did:
Heading, with name, version, description written by me/cribbed from the originals
Sources: The original places to download the Linux packages, official documentation, ...
%prep: Just unpack the different pieces, delete some redundant files, ...
%build: Nothing to do
%install: Create the relevant directories under $RPM_BUILD_ROOT by hand, copy files there by install, copy/create configuration files, ...
%clean: Blow away $RPM_BUILD_ROOT
%files: An exhaustive list of all files installed.
This required a few iterations to get right. Afterwards I followed the upstream package by rebuilding my RPM (conveniently I had everything packaged up in a SRPM, where the Source part was kind of a misnomer...)

How to "repackage" a RPM file for example cpio2rpm without installing the RPM?

I'm able to extract files from a RPM file, but how do I "rebuild" it, for example cpio2rpm?
I have extracted RPM file using following command.
rpm2cpio theFileName.rpm | cpio –idmv
I have to modify the few web application files like *.php, *.html or .js. These files don’t require any source recompilation. So I would like to replaces or change these files with modification without rebuilding rpm. Since, I need to do this for multiple platforms like Redhat Linux and SUSE, and multiple architecture like 32 and 64 bit OS.
I am expecting to do these changes on only on system and without rebuild rpm and there would not be have target system architecture dependency (like i386, 64).
I am not looking like command rpmbuild –rebuild the.src.rpm since, I don’t have source. I need to be rebuild binary .RPM file(not source .rpm)
I want to do this without source and platform or architecture independent and without using spec file if possible.
Any buddy, could you please suggest any solution or any free tools.
Thank you to all whoever spends time to read and reply to my thread.
You can use rpmrebuild to modify an actual rpm file (it doesn't need to be installed).
Most of the examples for this use complicated inline edit commands to modify known files in particular ways, but you can use a normal editor. I used this to fix a shell script in an rpm file that I didn't have the source for. Call the command as
rpmrebuild -ep theFileName.rpm
This puts you in an editor with the spec file for the RPM. The name of the file will be something like ~/.tmp/rpmrebuild.12839/work/spec.2. If you look in, in this example, ~/.tmp/rpmrebuild.12839/work, you will find all of the files used to make the RPM (in my case, the file was in root/usr/sbin within that directory). So, go to another window, cd to that directory, and edit any files you need to change.
When you have finished editing files, go back to the edit window with the spec file, make any changes you need to that file (I didn't have any, since I wasn't adding or deleting files), save the file, and say "y" to the "Do you want to continue" question. It will then build a new RPM file, and tell you where it has put it (in my case, in ~/rpmbuild/RPMS/x86_64/)
You can repackage an installed RPM (including modified files) using rpmrebuild. http://rpmrebuild.sourceforge.net/
Obviously your binaries (if any) would have to be platform/architecture independent to work on all the OS flavors you're hoping for, but it sounds like if they're just web files that shouldn't be a problem.
Principially you can pack everything you want into a RPM file. Just treat what you have as "source" and write a SPEC file which puts the data where the compiled binaries would normally go.
Concerning RPM, I consider "source" "what I have" and "binary" "what I need to run". Not very exact terminology, but it helps working with RPMs.
Your spec file looks like any other spec file, what concerns the parameters etc. But the code part is different:
[...]
%prep
# Here you either have nothing to do or you already unpack the cpio and possibly modify it.
# %build can be omitted
%install
[ "${buildroot}" != "/" ] && [ -d ${buildroot} ] && rm -rf ${buildroot};
# Here you can either unpack the cpio or copy the data unpacked in %prep.
# Be careful to put it into %{buildroot} or $RPM_BUILD_ROOT.

How is Paths_pandoc.hs generated?

In the project pandoc, Paths_pandoc is imported in Shared.hs. Paths_pandoc.hs is located in dist/build/autogen/. How is it generated and what does it do for pandoc.
It's a file that is generated by Cabal.
When you specify Data-files: in your .cabal file for your project, those files will be copied to a good location for "data files" on your system when you run cabal install. On Windows, this might be "C:\Program Files\Something" and on Linux it might be "/usr/share/something" (At least when you do a --system install).
Your code needs to know where the files were copied to, so Cabal generates that special module, which contains variables for the install paths that were used to copy the data files, so that your code can find the installed data files.
The module does also contain other information that Cabal provides for you, but the primary purpose of the module is what I just described.
See this blog post for more information.

Resources