I'm writing a library named Foo for an example.
The __init__.py file:
from .foo_exceptions import *
from .foo_loop import FooLoop()
main_loop = FooLoop()
from .foo_functions import *
__all__ = ['main_loop'] + foo_exceptions.__all__ + foo_functions.__all__
When installed, it can be used like this:
# example A
from Foo import foo_create, main_loop
foo_obj = foo_create()
main_loop().register(foo_obj)
or like this:
# example B
import Foo
foo_obj = Foo.foo_create()
Foo.main_loop().register(foo_obj)
I clearly prefer the example B approach. No name conflicts and the source of each external object is explicitely stated.
So much for introduction, now my question. Inside this library I need to import something from a different file. Again, I have several ways to do it. And the question is which style to prefer - C, D or E? Read below.
# example C
from . import foo_exceptions
raise foo_exceptions.FooError("fail")
or
# example D
from .foo_exceptions import FooError
raise FooError("fail")
or
# example E
from . import FooError
raise FooError("fail")
Approach C has the disadvantage, that importing a whole module instead of importing just a few required objects increases the chance of a cyclical import problem. Also consider this line:
from . import foo_exceptions, main_loop
It looks like an import of 2 symbols from one source, but it isn't. The former (foo_exceptions) is a module (.py file) in the current directory and the latter is an object defined in __init__.py.
That's why I'm not using style C and the question in its final form is: D or E (and why)?
(Thank you for reading this long question. All code fragments are examples only and may contain typos)
After the answer from alexanderlukanin:
EDIT1: corrected errors in init.py
NOTE1: foo_ prefixes are only to emphasize the relationship between objects
EDIT2: When importing an object which is not part of the library interface, style E is not usable. I think we have a winner: It's the from .module import symbol form.
Don't use old-style relative imports:
# Import from foo/foo_loop.py
# This DOES NOT WORK in Python 3
# and MAY NOT WORK AS EXPECTED in Python 2
from foo_loop import FooLoop
# This is reliable and unambiguous
from .foo_loop import FooLoop
Don't use asterisk import unless you really have to.
# Namespace pollution! Name clashes!
from .submodule import *
Don't use prefixes - you've got namespaces exactly for that purpose.
# Unpythonic
from foo import foo_something_create
foo_something_create()
# Pythonic
import foo.something
foo.something.create()
Your package's API must be well-defined. Your implementation must not be too tangled. The rest is a matter of taste.
# [C] This is good.
# Import order: __init__.py, exceptions.py
from . import exceptions
raise exceptions.FooError
# [D] This is also fine.
# Import order is the same as above,
# only name binding inside the current module is different.
from .exceptions import FooError
raise FooError
# [E] This is not as good because it adds one unnecessary level of indirection
# submodule.py -> __init__.py -> exceptions.py
from . import FooError
raise FooError
See also: Circular (or cyclic) imports in Python
Related
I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))
Use the imp module, or the more direct __import__() function.
You can use exec:
exec("import myapp.commands.%s" % command)
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
I am trying to auto document types with sphinx autodoc, napoleon and autodoc_typehints but I am having problems as it does not work with most of my types. I am using the deap package to do some genetic optimization algorithm, which makes that I have some very specific types I guess sphinx cannot handle.
My conf.py file looks like this:
import os
import sys
sys.path.insert(0, os.path.abspath('../python'))
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
'sphinx.ext.napoleon',
'sphinx_autodoc_typehints'
]
set_type_checking_flag = False
always_document_param_types = False
I have an Algo.rst file with:
.. automodule:: python.algo.algo
:members: crossover_worker,
test
and my python.algo.algo module looks like this (I've added a dummy test function to show it works whenever I have no special types specified):
# Type hinting imports
from config.config import Config
from typing import List, Set, Dict, NamedTuple, Union, Tuple
from types import ModuleType
from numpy import ndarray
from numpy import float64
from multiprocessing.pool import MapResult
from deap.tools.support import Logbook, ParetoFront
from deap.base import Toolbox
from pandas.core.frame import DataFrame
from deap import creator
...
def crossover_worker(sindices: List[creator.Individual, creator.Individual]) -> Tuple[creator.Individual, creator.Individual]:
"""
Uniform crossover using fixed threshold
Args:
sindices: list of two individuals on which we want to perform crossover
Returns:
tuple of the two individuals with crossover applied
"""
ind1, ind2 = sindices
size = len(ind1)
for i in range(size):
if random.random() < 0.4:
ind1[i], ind2[i] = ind2[i], ind1[i]
return ind1, ind2
def test(a: DataFrame, b: List[int]) -> float:
"""
test funcition
Args:
a: something
b: something
Returns:
something
"""
return b
When settings in conf.py are like above I have no error, types for my test function are correct, but types for my crossover_worker function are missing:
However, when I set the set_type_checking_flag= True to force using all types, I have a circular import error:
reading sources... [100%] index
WARNING: autodoc: failed to import module 'algo' from module 'python.algo'; the following exception was raised:
cannot import name 'ArrayLike' from partially initialized module 'pandas._typing' (most likely due to a circular import) (/usr/local/lib/python3.8/site-packages/pandas/_typing.py)
looking for now-outdated files... none found
And I never import ArrayLike so I don't get it from where it comes or how to solve it?
Or how to force to import also the creator.Individual types that appear everywhere in my code?
My sphinx versions:
sphinx==3.0.1
sphinx-autodoc-typehints==1.10.3
After some searching there were some flaws with my approach:
Firstly a "list is a homogeneous structure containing values of one type. As such, List only takes a single type, and every element of that list has to have that type." (source). Consequently, I cannot do something like List[creator.Individual, creator.Individual], but should transform it to List[creator.Individual] or if you have multiple types in the list, you should use an union operator, such as List[Union[int,float]]
Secondly, the type creator.Individual is not recognized by sphinx as a valid type. Instead I should define it using TypeVar as such:
from typing import TypeVar, List
CreatorIndividual = TypeVar("CreatorIndividual", bound=List[int])
So by transforming my crossover_worker function to this, it all worked:
def crossover_worker(sindices: List[CreatorIndividual]) -> Tuple[CreatorIndividual, CreatorIndividual]:
Note: "By contrast, a tuple is an example of a product type, a type consisting of a fixed set of types, and whose values are a collection of values, one from each type in the product type. Tuple[int,int,int], Tuple[str,int] and Tuple[int,str] are all distinct types, distinguished both by the number of types in the product and the order in which they appear."(source)
Using the AST in python 3, how do you build a directory or list of code dependencies of a given function?
Consider the following code, where my_clever_function has the desired behaviour:
////// myfile2.py
import numpy as np
a = 1
a += 1
def my_other_function():
def f():
return a
return np.random.randint() + f()
////// myfile1.py
import numpy as np
from . myfile2 import my_other_function
def external(a, b):
return np.sqrt(a * b) + my_other_function
class A:
def afunc(self, a, b):
v = external(a, b)
return v
>>> my_clever_function(A.afunc)
[myfile1.A.afunc, myfile1.external, myfile2.my_other_function, myfile2.a]
with the following structure:
project/
myfile1.py
myfile2.py
I want to retrieve the dependencies of the method afunc as a list.
I'm assuming that there is no funny business about functions altering global variables.
external is a dependency because it is not defined inside A.afunc
np.sqrt is not a "dependency" (in this sense anyway) because it is not defined in my project
likewise for np.random.randint
my_other_function is a dependency because it is not defined inside A.afunc
f is not a dependency because it is inside my_other_function
f needs the global variable a.
My motivation is to see if there have been any code changes between two project versions (in git perhaps).
We could find the dependencies of function like above and store the source.
In the future, we find the dependencies again and see if the source code is different.
We only compare the bits that are required (barring any funny global variables messing inside functions).
It is possible to walk the AST with python's builtin module ast.
So my_clever_function could look like this:
import ast
import dill
class Analyzer(ast.NodeVisitor):
def __init__(self):
self.stats = {...}
...
def report(self):
pprint(self.stats)
def my_clever_function(f):
source = dill.source.getsource(f)
tree = ast.parse(source)
analyser = Analyser()
analyser.visit(tree)
But how do you walk from a given function outwards to its dependencies?
I can see how you can just list symbols (https://www.mattlayman.com/blog/2018/decipher-python-ast/) but how do only list those which depend on the start node?
I'm trying to find out for myself how I could work around the problem I recently asked here and I come across a potential solution, but I honestly don't understand why it works, and why the other doesn't.
For context, model requires variables a and b to be defined before being successfully loaded and defined in a module. Otherwise, it throws an error: NameError: name 'a' is not defined.
Starting off with model.py:
import pickle
from tensorflow import keras
# loads and returns the variables needed by model
def load_model_vars():
return pickle.load(open('./file.pkl', 'rb'))
# loads and returns the model
def load_model():
return keras.models.load_model('./model.h5')
Now to minimally reproduce and identify the problem I created a new module, foo.py:
from model import load_model_vars, load_model
# goal here is to supposedly expose only the model to other modules
a, b = load_model_vars()
globals()['model'] = load_model()
I then created another module to import foo.py into, let's name it bar.py:
import foo
# just checks if the model is defined
foo.model.summary()
Which for some reason throws the formerly mentioned NameError. Why? The variables are defined, it was executed in order(load variables first, then model), and even if I change a, b to globals()['a'], globals()['b'], import foo to from foo import * or from foo import a, b or even combinations of any of these, it always arrives into this error.
But when I introduce another module, say, baz.py, that contains these two lines:
from model import load_model_vars
a, b = load_model_vars()
Then import it to bar.py:
from baz import a, b
import foo
# just checks if the model is defined
foo.model.summary()
With foo.py unchanged, or with a, b = load_model_vars() commented out:
from model import load_model_vars, load_model
# goal here is to supposedly expose only the model to other modules
# a, b = load_model_vars()
globals()['model'] = load_model()
It successfully loads the freaking model! Why? What's this sorcery underneath the import function? What actually happens under the hood?
I wanted to use import with a variable name. For example I wanted to do something like this
from var import my_class
I went through pythons documentation, but seems thats a little confusing. Also I seen some other posting on stack overflow that give the example of something like this
import importlib
my_module = importlib.import_module("var, my_class)
This second example does work to a certain extent. The only issue I see here var is imported but I don't see the attributes of my_class in python's namespace. How would I equate this to my original example of
from var import my_class
Here's how to use importlib (there is no need for the second parameter):
var = importlib.import_module("var")
# Now, you can use the content of the module:
var.my_class()
There is no direct programmable equivalent for from var import my_class.
Note: As #DYZ points out in the comments, this way of solving this is not recommended in favor of importlib. Leaving it here for the sake of another working solution, but the Python docs advise "Direct use of import() is also discouraged in favor of importlib.import_module()."
Do you mean that you want to import a module whose name is defined by a variable? If so, you can use the __import__ method. For example:
>>> import os
>>> os.getcwd()
'/Users/christophershroba'
>>>
>>> name_to_import = "os"
>>> variable_module = __import__(name_to_import)
>>> variable_module.getcwd()
'/Users/christophershroba'
If you also want to call a variable method of that variable module you could use the __getattribute__ method on the module to get the function, and then call it with () as normal. The line below marked "See note" is not necessary, I just include it to show that the __getattribute__ method is returning a function.
>>> name_to_import = "os"
>>> method_to_call = "getcwd"
>>> variable_module = __import__(name_to_import)
>>> variable_module.__getattribute__(method_to_call) # See note
<built-in function getcwd>
>>> variable_module.__getattribute__(method_to_call)()
'/Users/christophershroba'
More documentation available for Python 3 here or Python2 here.