I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))
Use the imp module, or the more direct __import__() function.
You can use exec:
exec("import myapp.commands.%s" % command)
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
Related
I wan to import all the functions and class into a module/file of Python in high level file just passing a variable that contains the low level file name.
I have a application with several module like:
__all__ = ['MyClass1', 'my_function1']
class MyClass1():
pass
def my_function1():
pass
that previous was import at the high level file as:
from sub_module1 import *
from sub_module2 import *
...
# To direct use, of the different subfiles:
obj1 = MyClass1()
obj2 = MyClass2()
The application became a plugin based and I have to dynamic import all module into a folder and provide direct access to all objects defined into __all__ of those submodules.
The code bellow imports fine the submodules but I don not give my direct access to the directives defined into __all__ of those files.
from os import path
from importlib import import_module
directory_name = ## Define the plugins dir.
for importer, package_name, _ in iter_modules([directory_name]):
module_specification = importlib.util.spec_from_file_location(
package_name, path.join(directory_name, package_name + '.py'))
module_loader = importlib.util.module_from_spec(module_specification)
module_specification.loader.exec_module(module_loader)
How do I put those object define into __all__ of the submodules inside locals() of the high module?
I am attempting to run timeit.timeit in the following class:
from contextlib import suppress
from pathlib import Path
import subprocess
from timeit import timeit
class BackupVolume():
'''
Backup a file system on a volume using tar
'''
targetFile = "bd.tar.gz"
srcPath = Path("/BulkData")
excludes = ["--exclude=VirtualBox VMs/*", # Exclude all the VM stuff
"--exclude=*.tar*"] # Exclude this tar file
#classmethod
def backupData(cls, targetPath="~"): # pylint: disable=invalid-name
'''
Runs tar to backup the data in /BulkData so we can reorganize that
volume. Deletes any old copy of the backup repository.
Parameters:
:param str targetPath: Where the backup should be created.
'''
# pylint: disable=invalid-name
tarFile\
= Path(Path(targetPath /
cls.targetFile).resolve())
with suppress(FileNotFoundError):
tarFile.unlink()
timeit('subprocess.run(["tar", "-cf", tarFile.as_posix(),'
'cls.excludes[0], cls.excludes[1], cls.srcPath.as_posix()])',
number=1, globals=something)
The problem I have is that inside timeit() it cannot interpret subprocess. I believe that the globals argument to timeit() should help but I have no idea how to specify the module namespace. Can someone show me how?
I think in your case globals = globals() in the timeit call would work.
Explanation
The globals argument specifies a namespace in which to execute the code. Due to your import of the subprocess module (outside the function, even outside the class) you can use globals(). In doing so you have access to a dictionary of the current module, you can find more info in the documentation.
Super simple example
In this example I'll expose 3 different scenarios.
Need to access globals
Need to access locals
Custom namespace
Code to follow the example:
import subprocess
from timeit import timeit
import math
class ExampleClass():
def performance_glob(self):
return timeit("subprocess.run('ls')", number = 1, globals = globals())
def performance_loc(self):
a = 69
b = 42
return timeit("a * b", number = 1, globals = locals())
def performance_mix(self):
a = 69
return timeit("math.sqrt(a)", number = 1, globals = {'math': math, 'a': a})
In performance_glob you are timing something that needs a global import, the module subprocess. If you don't pass the globals namespace you'll get an error message like this NameError: name 'subprocess' is not defined
On the contrary, if you pass globals() to the function that depends on local values performance_loc the needed variables for the timeit execution a and b won't be in the scope. That's why you can use locals()
The last one is a general scenario where you need both the local vars in the function and general imports. If you keep in mind that the parameter globals can be specified as a dictionary, you just need to provide the necessary keys, you can customize it.
I wanted to use import with a variable name. For example I wanted to do something like this
from var import my_class
I went through pythons documentation, but seems thats a little confusing. Also I seen some other posting on stack overflow that give the example of something like this
import importlib
my_module = importlib.import_module("var, my_class)
This second example does work to a certain extent. The only issue I see here var is imported but I don't see the attributes of my_class in python's namespace. How would I equate this to my original example of
from var import my_class
Here's how to use importlib (there is no need for the second parameter):
var = importlib.import_module("var")
# Now, you can use the content of the module:
var.my_class()
There is no direct programmable equivalent for from var import my_class.
Note: As #DYZ points out in the comments, this way of solving this is not recommended in favor of importlib. Leaving it here for the sake of another working solution, but the Python docs advise "Direct use of import() is also discouraged in favor of importlib.import_module()."
Do you mean that you want to import a module whose name is defined by a variable? If so, you can use the __import__ method. For example:
>>> import os
>>> os.getcwd()
'/Users/christophershroba'
>>>
>>> name_to_import = "os"
>>> variable_module = __import__(name_to_import)
>>> variable_module.getcwd()
'/Users/christophershroba'
If you also want to call a variable method of that variable module you could use the __getattribute__ method on the module to get the function, and then call it with () as normal. The line below marked "See note" is not necessary, I just include it to show that the __getattribute__ method is returning a function.
>>> name_to_import = "os"
>>> method_to_call = "getcwd"
>>> variable_module = __import__(name_to_import)
>>> variable_module.__getattribute__(method_to_call) # See note
<built-in function getcwd>
>>> variable_module.__getattribute__(method_to_call)()
'/Users/christophershroba'
More documentation available for Python 3 here or Python2 here.
I have a boost python generating a shared object to be used with python in /home/user/service/org/boost_py.so (This folder does not contain a __init__.py)
and /home/user/service is part of sys.path so when I need to use this ,just do
import org.boost_py #works
Now I have added a pure python module in a different directory.
/home/user/service/pure_python/org/
__init__.py
tester.py
__init__.py contains
__import__('pkg_resources').declare_namespace(__name__)
Now when sys.path is
['/home/user/service/','/home/user/service/pure_python']
and I
import org.boost_py #ImportError: No module named 'org.boost_py'
but I can import org.tester. How to import both org.tester and org.boost_py ? (I cannot change the location of either)
update:
Found pth file with following contents in pure_python directory
import sys, types, os;
p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('org',));
ie = os.path.exists(os.path.join(p,'__init__.py'));
m = not ie and sys.modules.setdefault('org', types.ModuleType('org'));
mp = (m or []) and m.__dict__.setdefault('__path__',[]);
(p not in mp) and mp.append(p)
Apparently there is no file or directory boost_py in 'pure_python/org ', this produces the error.
If it exists please edit the question!
Alternatively, the path of the boost could be not in the path browser, then it just wont recognize the file.
from foo.bar import bar
print(get_namespace(bar))
I want the above code snippet to print the "foo.bar.bar"
How to do it in python3 ?
Every imported module is listed in sys.modules. So you could iterate through sys.modules to find the path associated with the module:
import sys
def get_module_name(mod):
for path, module in sys.modules.items():
if mod == module:
return path
raise ValueError('module not found')
from foo.bar import bar
print(get_module_name(bar))
# foo.bar.bar
Since the word namespace is reserved for a different concept, I've renamed the function get_module_name.