I've come across some code recently that uses a trick that makes me rather nervous. The system I'm looking at has a Python extension Moo.so file stored outside the path and the developer wants to import it with just import Moo. For various reasons neither the file location nor sys.path can be changed, and the extension must be loaded with ExtensionFileLoader anyway.
So what has been done is to have a Moo.py in the path that loads the extension module and then replaces itself in sys.modules with the extension module, along the following lines:
' Moo.py '
from importlib.machinery import ExtensionFileLoader
loader = ExtensionFileLoader('AnotherNameForMoo', '/path/to/Moo.so')
module = loader.load_module()
sys.modules['Moo'] = module
Now this does actually work. (I have some tests of it in rather gory detail in this repo if you want to have a look.) It appears that, at least in CPython 3.4 through 3.7, import Moo does not bind to Moo the module that it loaded and put into sys.modules['Moo'], but instead binds the current value of sys.modules['Moo'] after the module's top-level script returns, regardless of whether or not that's what it originally put in there.
I can't find anything in any Python documentation that indicates that this is required behaviour rather than just an accident of implementation.
How safe is this? What are other ways that one might try to achieve a similar "bootstrap" effect?
Related
I'm trying out Freeling's API for python. The installation and test were ok, they provide a sample.py file that works perfectly (I've played around a little bit with it and it works).
So I was trying to use it on some other python code I have, in a different folder (I'm kind of guessing this is a path issue), but whenever I import freeling (like it shows on the sample.py):
import freeling
FREELINGDIR = "/usr/local";
DATA = FREELINGDIR+"/share/freeling/";
LANG="es";
freeling.util_init_locale("default");
I get this error:
ModuleNotFoundError: No module named 'freeling'.
The sample.py is located on the ~/Freeling-4.0/APIs/Python/ folder, while my other file is located in ~/project/, I dont know if that can be an issue.
Thank you!
A simple solution is to have a copy of freeling.py in the same directory as your code, since python will look there.
A better solution is to either paste it in one of the locations where it usually checks (like the lib folder in its install directory), or to tell it that the path where your file is should be scanned for a module.
You can check out this question to see how it can be done on Windows. You are basically just setting the PYTHONPATH environment variable, and there will only be minor differences in how to do so for other OSes. This page gives instructions that should work on Linux systems.
I like this answer since it adds the path at runtime in the script itself, doesn't make persistent changes, and is largely independent of the underlying OS (apart from the fact that you need to use the appropriate module path of course).
You need to set PYTHONPATH so python can find the modules if they are not in the same folder.
I am trying to create a Python package. You can have a look at my awful attempt here.
I have a module called imguralbum.py. It lives in a directory called ImgurAlbumDownloader which I understand is the name of the package -- in terms of what you type in an import statement, e.g.
import ImgurAlbumDownloader
My module contains two classes ImgurAlbumDownloader and ImgurAlbumException. I need to be able to use both of these classes in another module (script). However, I cannot for the life of me work out what I am supposed to put in my __init__.py file to make this so. I realize that this duplicates a lot of previously answered questions, but the advice seems very conflicting.
I still have to figure out why (I have some ideas), but this is now working:
from ImgurAlbumDownloader.imguralbum import ImgurAlbumDownloader, ImgurAlbumException
The trick was adding the package name to the module name.
It sure sounds to me like you don't actually want a package. It's OK to just use a single module, if your code just does one main thing and all its parts are closely related. Packages are useful when you have distinct parts of your code that might not all be needed at the same time, or when you have so much code that a single module would be very large and hard to find things in.
Suppose we have the following project structure:
E:\demo_proj\
utilities_package\
__init__.py
util_one.py
util_two.py
demo_package\
__init__.py
demo_sub_package\
__init__.py
demo_sub_sub_package\
demo_file.py
What is a sensible way for demo_file.py to import from utilities_package?
The utilities_package is in a much higher level directory than demo_file.py.
Although it is not shown in the little diagram I gave above, suppose that utilities_package is used everywhere throughout the project. It is not sensible to make utilities_package a sub-directory of demo_sub_sub_package
Some similar questions have been asked here and here.
The asker of the first question neglected to include a __init__.py in their packages. That is not an issue in my case.
The answers given to the the second question linked-to above are problematic:
Most of the answers are quite old. For example, Loader.load_module was deprecated after Python 3.4
Most of the answers are specific to the question in that they only go up one directory, whereas, I would like to go up several levels.
Some of the answers are high-level descriptions without code. Other answers have the opposite problem: they contain ugly-looking code, but don't bother to explain it.
I am open to solutions which require restructuring the project. I'm also interested in solutions which leave the project structure intact, but which insert the relevant code into demo_file.py
It already has the very epic solution for your problems. Please refer there: Relative imports for the billionth time. Just as a notation, the python3 and python2 have the totally different definitions for the relative and absolute import. For version 3+, any modules import directly using import module_name are treated as the absolute importation, which means this module must be included in the sys.path. Otherwise, for any modules import like from one_module import something or from . import something are treated like the relative importation. The dot-notation represent the current directory. However, any module is executed directly by itself, it will lose its namespace, instead it will always be treated as __main__, and all the files in the same directory will automatically add to sys.path by default. Hope it will help you.
I have written a module for teaching Python. I'd like to make it difficult for the smarter ones to view the source code as a short-cut. Does not need to be fully secure - disabling the inspect module might be enough - if this is possible.
In case this is useful to anyone else using Python3 for class tests etc here's what I've ended up doing (with thanks to wbwlkr).
python3 -OO -m py_compile testmod.py creates a file __pycache__/testmod.cpython-34.pyo
Creating a symbolic link to this file named testmod.pyc means the code can't easily be inspected.
One other consideration is that sensitive local variables should be overwritten when not needed or they can be queried by locals()
What you are searching for is a solution to "obfuscate" the source code of your module.
You could compile your module to byte-code, as suggested here :
https://stackoverflow.com/a/7418341/8714367
Is there a way to have custom behaviour for import statements in Python? How? E.g.:
import "https://github.com/kennethreitz/requests"
import requests#2.11.1
import requests#7322a09379565bbeba9bb40000b41eab8856352e
Alternatively, in case this isn't possible... Can this be achieved in a standard way with function calls? How? E.g.:
import gitloader
repo = gitloader.repo("https://github.com/kennethreitz/requests")
requests = repo.from_commit("7322a09379565bbeba9bb40000b41eab8856352e")
There are two reasons for why I would like to do this. The first reason is convenience (Golang style imports). The second reason is that I need cryptographic verification of plugin modules for a project. I'm using Python 3.x
What you're essentially asking is customization of the import steps. This was made possible with PEP 302 which brought about certain hooks for for customization.
That PEP is currently not the source from which you should learn how importing works; rather, look at the documentation for the import statement for reference. There it states:
Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an import path for modules. The import path is a list of locations that may name file system paths or zip files. It can also be extended to search for any locatable resource, such as those identified by URLs.
The import machinery is extensible, so new finders can be added to extend the range and scope of module searching.
In short, you'll need to define the appropriate finder (to find the modules you're looking for) and an appropriate loader for loading these.