I am taking a close look at Scons and something smells. SCons uses SConstruct files as base configuration file. This configuration file is a Python file but:
It does not have the .py extension
It does not have any import directives
It is not possible to have auto-completion from IDEs
It it possible to use a variant of the SConstruct file where I could find something like the following?
# build.py
import scons
env = scons.Environment()
env.Program('foo')
It would be not simple (but possible) to do what you're asking. SConscripts are plain python, however the globals available in the context of SConstruct or SConscripts are carefully constructed.
Any user can add methods and also pass python objects into the SConscripts via Export() or exports (in a SConscript call).
That said try:
from SCons.Script import *
That should get you some of what you're looking for.
The fact that Pycharm cannot find the symbols in question doesn't mean the it is not a plain python file.
Additionally I'm not sure how the subject of your question relates to the contents of your question. Typically setup.py is a file to build packages ala setuptools and install via pip (or similar).
Sure you can build whatever you are trying to build with setuptools, it will likely be harder to do, but that said if you get it to work, perhaps easier to upload and distribute via pypi
p.s. It's SCons not Scons.
Related
I want to create a git repo that can be used like this:
git clone $PROJECT_URL my_project
cd my_project
python some_dir/some_script.py
And I want some_dir/some_script.py to import from another_dir/some_module.py.
How can I accomplish this?
Some desired requirements, in order of decreasing importance to me:
No sys.path modifications from within any of the .py files. This leads to fragility when doing IDE-powered automated refactoring.
No directory structure changes. The repo has been thoughtfully structured.
No changes to my environment. I don't want to add a hard-coded path to my $PYTHONPATH for instance, as that can result in unexpected behavior when I cd to other directories and launch unrelated python commands.
Minimal changes to the sequence of 3 commands above. I don't want a complicated workflow, I want to use tab-completion for some_dir/some_script.py, and I don't want to spend keystrokes on extra python cmdline flags.
I see four solutions to my general problem described here, but none of them meet all of the above requirements.
If no solution is possible, then why are things this way? This seems like such a natural want, and the requirements I list seem perfectly reasonable. I'm aware of a religious argument in a 2007 email from Guido:
I'm -1 on this and on any other proposed twiddlings of the __main__
machinery. The only use case seems to be running scripts that happen
to be living inside a module's directory, which I've always seen as an
antipattern. To make me change my mind you'd have to convince me that
it isn't.
But not sure if things have changed since then.
Opinions haven't changed on this topic since Guido's 2007 comment. If anything, we're moving even further in the opposite direction, with the additions of PYTHONSAFEPATH var and corresponding -P option in 3.11:
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSAFEPATH
https://docs.python.org/3/using/cmdline.html#cmdoption-P
These options will nerf direct sibling module imports too, requiring sys.path to be explicitly configured even for scripts!
So, scripts still can't easily do relative imports, and executable scripts living within a package structure are still considered an anti-pattern. What to do instead?! The widely accepted alternative here is to use the packaging feature of entry-points. One type of entry-point group in packaging metadata is the "console_scripts" group, used to point to arbitrary callables defined within your package code. If you add entries in this group within your package metadata, then script wrappers for those callables will be auto-generated and put somewhere on $PATH at pip install time). No hacking of sys.path necessary.
That being said, it's still possible to run .py files directly as scripts, provided you've configured the underlying Python environment for them to resolve their dependencies (imports) correctly. To do that, you'll want to define a package structure and "install" the package so that your source code is visible on sys.path.
Here's a minimum example:
my_project
├── another_dir
│ ├── __init__.py <-- __init__ file required for package dirs (it can be empty)
│ └── some_module.py
├── pyproject.toml <-- packaging metadata lives here
└── some_dir <-- no __init__ file necessary for non-packaged subdirs
└── some_script.py
Minimal contents of the packaging definition in pyproject.toml:
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my_proj"
version = "0.1"
[tool.setuptools.packages.find]
namespaces = false
An additional once-off step is required to create/configure an environment in between the git clone and the script execution:
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
This makes sure that another_dir is available to import from the environment's site-packages directory, which is already one of the locations on sys.path (check with python -m site). That's what's required for any/all of these import statements to work from within the script file(s)
from another_dir import some_module
import another_dir.some_module
from another_dir.some_module import something
Note that this does not necessarily put the parent of another_dir onto sys.path directly. For an editable install, it will setup some scaffolding which makes your package appear to be "installed" in the site, which is sufficient for those imports to succeed. For a non-editable install (pip install without the -e flag), it will just copy your package directly into the site, compile the .pyc files, and then the code will be found by the normal SourceFileLoader.
I'm trying to build an executable from another external SCons project to use it in my SCons build.
At first I tried to run:
Command('other_project_dir/executable', None, 'scons -C other_project_dir')
This doesn't work because SCons by default doesn't use the system PATH variable.
Even if it would work, another issue would be that is someone called my script with a custom path to SCons. The other script would still be built with the default SCons.
In conclusion, I need find out what is the path of the SCons that is currently running.
Is there any method to get it, or an environment variable that stores it?
Currently (in version 4.3.0) SCons doesn't have a built in way to obtain its path.
However, you don't really need any spacial functions. You can just use plain Python:
import sys
scons_path = sys.argv[0]
Command('other_project_dir/executable', scons_path, '$SOURCE -C other_project_dir')
Don't try to use the variable __file__.
It is purposely removed by SCons.
BTW, you should run AlwaysRun('other_project_dir/executable').
Other way SCons may not rebuild the other project when it's necessary since it doesn't aware about the dependencies in the other project.
We are using the wheels to deploy our code to QA/Production. Recently we found/realized that wheel packages are actually storing our source code. And by simple command as below will open all the source code inside it.
unzip package.whl
command used for wheel creation is as below
cd /path/to/source/code/folder
python setup.py bdist bdist_wheel
So,
Is there any way to create wheels which creates binary and stores in package rather than source code?
In the simplest sense, wheel is just:
a zip file
with a specific filename
and a specific directory layout
containing pure-Python source code
and any platform-specific binaries
This means that a wheel (and any other distribution) is not a binary itself, but it may contain platform-specific binaries -- for example, if you are building/compiling some C code along with your Python package.
Most wheels are pure-Python, which means that they only contain Python source code.
It seems like you're asking how to "compile" Python code into an obfuscated binary. This is not the goal of a wheel. You might want to read more details on the wheel format here: https://www.python.org/dev/peps/pep-0427/
Is there any way to create wheels which creates binary and stores in package rather than source code?
Not with the wheel format. If this is actually your goal, you may want to look into pyinstaller, py2exe or cython, depending on the target platform.
In a case someone stumbles here the same way I did. If you
Use Cython to pre-compile your library.
Have not only .pyx, but also .py modules. For example you want to do this with some existing project without any modifications (except for setup.py), or consider it unreasonable to do in the first place as Cython consumes .py files also).
Want to distribute pre-compiled library without any .py files included (except for may be empty __init__.py files).
Then, you can apply the following (quite dirty) solution to exclude any files you want from the wheel:
from wheel.bdist_wheel import bdist_wheel
class CommandBdistWheel(bdist_wheel):
# Called almost exactly before filling `.whl` archive
def write_wheelfile(self, *args, **kwargs):
dr = f'{self.bdist_dir}/<package name>'
paths = [
path for path in glob.glob(f'{dr}/**/*.py', recursive=True)
if os.path.basename(path) != '__init__.py'
]
for path in paths:
os.remove(path)
super().write_wheelfile(*args, **kwargs)
setup(
# ...
cmdclass={'bdist_wheel': CommandBdistWheel},
# ...
)
condensed version of what I want to achieve:
Create .rpm and .deb packages from my source.py source code and make sure all dependencies get resolved when installing them on an deb/rpm based linux distribution.
More details:
Let's assume I have created a piece of software which is located in a folder structure like this:
---MyProgram Folder
---MyProgram Folder
---img Folder
---logo.ico File
---media Folder
---head.txt File
---__init__.py File
---source.py File
---a.py File
---LICENSE File
---README.md File
---setup.py File
The file setup.py contains the following:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="MyProgram",
version="0.0.1",
author="First Last",
author_email="email#memore.com",
description="A tool to create nice things",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://google.com",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.7',
data_files=[
('.../MyProgram/img/logo.ico'),
('.../MyProgram/media/head.txt'),
],
)
I now run
python setup.py sdist bdist_rpm
from a cmd line under '.../MyProgram'. Two folders 'dist' and 'build' are created as well as 'MyProgram.tar.gz' and two rpm's 'MyProgram-noarch.rpm' and 'MyProgram-src.rpm'.
When i try to install 'noarch.rpm' under fedora 31 the process end successfully but no "shortcut" is created, and when i type MyProgram in a cmd line it is not found.
rpm -ql MyFilter
does find it and outputs a bunch of paths:
/usr/lib/python3.7/site-packages/MyProgram/...
/usr/lib/python3.7/site-packages/MyProgram/source.py
/usr/lib/python3.7/site-packages/MyProgram/a.py
....
Which tells me that my installation at least has copied the basic filesystem. But i also see that all the original .py files are still .py files.
My questions:
How can i 'make' the rpm so that all dependencies are contained inside the rpm, or at least get resolved by dnf/apt/yum when installing the rpm? In other wording: Is it possible to bundle all dependencies into a rpm/deb like in an .exe for example?
How can i specify a path like '/usr/bin' or 'usr/share' as installation target
dir?
How can i add a launcher app bundled into the rpm/deb?
Is the above a good way of doing this at all?
If the solution to this is trivial and i just overlooked it i am really sorry to bother you but atm i just can't see it. Sites that have relevant information and that i already reviewed:
https://docs.python.org/2.0/dist/creating-rpms.html
https://github.com/AppImage/AppImageKit/wiki/Bundling-Python-apps
Python 3.5 create .rpm with pyinstaller generated executable
https://github.com/junaruga/rpm-py-installer
https://www.pyinstaller.org/
https://packaging.python.org/overview/#python-source-distributions
https://packaging.python.org/overview/
https://pyinstaller.readthedocs.io/en/stable/usage.html
https://pyinstaller.readthedocs.io/en/stable/installation.html
https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html
Just my two cents, rather than a complete answer. Will mostly touch on RPM packaging.
The bdist_rpm option seems easy, but you have little control of the logic of the .spec file it generates/uses and cannot do fancy stuff like scriplets, etc.
That is, unless you take the approach of having it generate the .spec file and quit (instead of building final RPM). From the docs:
If you wish, you can separate these three steps. You can use the --spec-only option to make bdist_rpm just create the .spec file and exit; in this case, the .spec file will be written to the “distribution directory”—normally dist/, but customizable with the --dist-dir option. (Normally, the .spec file winds up deep in the “build tree,” in a temporary directory created by bdist_rpm.)
But as a matter of preference and consistency, I would advise on following distro-specific guidelines for packaging Python apps.
In that way, you will be more in line with the distro's you are building for.
It is not the easiest way though. You will have to shift through some docs. Basically, if you're building for anything CentOS/RHEL, Fedora guidelines for packaging should be observed.
You can find the extra reference here, with the example .spec file for building both Python 2 and 3 versions of the same app.
For this whole 'build like a distro' thing, you would definitely want to look into using mock for the job, to build your package in a chroot.
As for the "shortcut" issue, you have to have your setup.py declare some console scripts for it to create one when you install your package. E.g. from lastversion's setup.py:
entry_points={"console_scripts": ["lastversion = lastversion:main"]},
This entry will result in a "binary" lastversion created/installed (which runs the defined function) when you install your Python package.
Subsequently, in the spec files, the macro %py2_install will make use of setup.py to create the same launcher program.
And you will then be able to ensure that launcher is packaged by placing it in the files section of the spec file:
%files -n python3-myapp
%license COPYING
%doc README.rst
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}-*.egg-info/
%{_bindir}/myapp
I have some scripts in my package, that rely on some template xml files.
Those scripts are callable by entry points and I wanted to reference the template files by a relative path.
When calling the script via python -m ... the scripts themselves are called from within lib\site-packages and there the xml files are available as I put them in my setup.py like this:
setup(
...
packages=['my_pck'],
package_dir={'my_pck': 'python/src/my_pck'},
package_data={'my_pck': ['reports/templates/*.xml']},
...
)
I know, I could copy those templates also by using data_files in my setup.py but using package_data seems better to me.
Unfortunately package_data seems not to copy those files to the Scripts folder where the entry points are located.
So my question is, is this even achievable via package_data and if, how?
Or is there a more pythonic, easier way to achieve this? Maybe not referencing those files via paths relative to the scripts?
Looks like importlib-resources might help here. This library is able to find the actual path to a resource file packaged as package_data by setuptools.
Access the package_data files from your code with something like this:
with importlib_resources.path('my_pck.reports.templates', 'a.xml') as xml_path:
do_something(xml_path)