Related
I wan to import all the functions and class into a module/file of Python in high level file just passing a variable that contains the low level file name.
I have a application with several module like:
__all__ = ['MyClass1', 'my_function1']
class MyClass1():
pass
def my_function1():
pass
that previous was import at the high level file as:
from sub_module1 import *
from sub_module2 import *
...
# To direct use, of the different subfiles:
obj1 = MyClass1()
obj2 = MyClass2()
The application became a plugin based and I have to dynamic import all module into a folder and provide direct access to all objects defined into __all__ of those submodules.
The code bellow imports fine the submodules but I don not give my direct access to the directives defined into __all__ of those files.
from os import path
from importlib import import_module
directory_name = ## Define the plugins dir.
for importer, package_name, _ in iter_modules([directory_name]):
module_specification = importlib.util.spec_from_file_location(
package_name, path.join(directory_name, package_name + '.py'))
module_loader = importlib.util.module_from_spec(module_specification)
module_specification.loader.exec_module(module_loader)
How do I put those object define into __all__ of the submodules inside locals() of the high module?
I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))
Use the imp module, or the more direct __import__() function.
You can use exec:
exec("import myapp.commands.%s" % command)
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
Currently I am extracting files to the temp directory of the operating system. One of the files is a Python file containing a class which I need to get a handle of. The Python's file is known, but the name of the class inside the file is unknown. But it is safe to assume, that the there is only one single class, and that the class is a subclass of another.
I tried to work with importlib, but I am not able to get a handle of the class.
So far I tried:
# Assume
# module_name contains the name of the class and -> "MyClass"
# path_module contains the path to the python file -> "../Module.py"
spec = spec_from_file_location(module_name, path_module)
module = module_from_spec(spec)
for pair in inspect.getmembers(module):
print(f"{pair[1]} is class: {inspect.isclass(pair[1])}")
When I iterate over the members of the module, none of them get printed as a class.
My class in this case is called BasicModel and the Output on the console looks like this:
BasicModel is class: False
What is the correct approach to this?
Edit:
As the content of the file was requested, here you go:
class BasicModel(Sequential):
def __init__(self, class_count: int, input_shape: tuple):
Sequential.__init__(self)
self.add(Input(shape=input_shape))
self.add(Flatten())
self.add(Dense(128, activation=nn.relu))
self.add(Dense(128, activation=nn.relu))
self.add(Dense(class_count, activation=nn.softmax))
Use dir() to get the attributes of the file and inspect to check if the attribute is a class. If so, you can create an object.
Assuming that your file's path is /tmp/mysterious you can do this:
import importlib
import inspect
from pathlib import Path
import sys
path_pyfile = Path('/tmp/mysterious.py')
sys.path.append(str(path_pyfile.parent))
mysterious = importlib.import_module(path_pyfile.stem)
for name_local in dir(mysterious):
if inspect.isclass(getattr(mysterious, name_local)):
print(f'{name_local} is a class')
MysteriousClass = getattr(mysterious, name_local)
mysterious_object = MysteriousClass()
How to make this code works?
There is a zip file with folders and .png files in it. Folder ".\icons_by_year" is empty. I need to get every file one by one without unzipping it and copy to the root of the selected folder (so no extra folders made).
class ArrangerOutZip(Arranger):
def __init__(self):
self.base_source_folder = '\\icons.zip'
self.base_output_folder = ".\\icons_by_year"
def proceed(self):
self.create_and_copy()
def create_and_copy(self):
reg_pattern = re.compile('.+\.\w{1,4}$')
f = open(self.base_source_folder, 'rb')
zfile = zipfile.ZipFile(f)
for cont in zfile.namelist():
if reg_pattern.match(cont):
with zfile.open(cont) as file:
shutil.copyfileobj(file, self.base_output_folder)
zfile.close()
f.close()
arranger = ArrangerOutZip()
arranger.proceed()
shutil.copyfileobj uses file objects for source and destination files. To open the destination you need to construct a file path for it. pathlib is a part of the standard python library and is a nice way to handle file paths. And ZipFile.extract does some of the work of creating intermediate output directories for you (plus sets file metadata) and can be used instead of copyfileobj.
One risk of unzipping files is that they can contain absolute or relative paths outside of the target directory you intend (e.g., "../../badvirus.exe"). extract is a bit too lax about that - putting those files in the root of the target directory - so I wrote a little something to reject the whole zip if you are being messed with.
With a few tweeks to make this a testable program,
from pathlib import Path
import re
import zipfile
#import shutil
#class ArrangerOutZip(Arranger):
class ArrangerOutZip:
def __init__(self, base_source_folder, base_output_folder):
self.base_source_folder = Path(base_source_folder).resolve(strict=True)
self.base_output_folder = Path(base_output_folder).resolve()
def proceed(self):
self.create_and_copy()
def create_and_copy(self):
"""Unzip files matching pattern to base_output_folder, raising
ValueError if any resulting paths are outside of that folder.
Output folder created if it does not exist."""
reg_pattern = re.compile('.+\.\w{1,4}$')
with open(self.base_source_folder, 'rb') as f:
with zipfile.ZipFile(f) as zfile:
wanted_files = [cont for cont in zfile.namelist()
if reg_pattern.match(cont)]
rebased_files = self._rebase_paths(wanted_files,
self.base_output_folder)
for cont, rebased in zip(wanted_files, rebased_files):
print(cont, rebased, rebased.parent)
# option 1: use shutil
#rebased.parent.mkdir(parents=True, exist_ok=True)
#with zfile.open(cont) as file, open(rebased, 'wb') as outfile:
# shutil.copyfileobj(file, outfile)
# option 2: zipfile does the work for you
zfile.extract(cont, self.base_output_folder)
#staticmethod
def _rebase_paths(pathlist, target_dir):
"""Rebase relative file paths to target directory, raising
ValueError if any resulting paths are not within target_dir"""
target = Path(target_dir).resolve()
newpaths = []
for path in pathlist:
newpath = target.joinpath(path).resolve()
newpath.relative_to(target) # raises ValueError if not subpath
newpaths.append(newpath)
return newpaths
#arranger = ArrangerOutZip('\\icons.zip', '.\\icons_by_year')
import sys
try:
arranger = ArrangerOutZip(sys.argv[1], sys.argv[2])
arranger.proceed()
except IndexError:
print("usage: test.py zipfile targetdir")
I'd take a look at the zipfile libraries' getinfo() and also ZipFile.Path() for construction since the constructor class can also use paths that way if you intend to do any creation.
Specifically PathObjects. This is able to do is to construct an object with a path in it, and it appears to be based on pathlib. Assuming you don't need to create zipfiles, you can ignore this ZipFile.Path()
However, that's not exactly what I wanted to point out. Rather consider the following:
zipfile.getinfo()
There is a person who I think is getting at this exact situation here:
https://www.programcreek.com/python/example/104991/zipfile.getinfo
This person seems to be getting a path using getinfo(). It's also clear that NOT every zipfile has the info.
File "C:\Users\Administrator\Documents\Mibot\oops\blinkserv.py", line 82, in __init__
self.serv = socket(AF_INET,SOCK_STREAM)
TypeError: 'module' object is not callable
Why am I getting this error?
I'm confused.
How can I solve this error?
socket is a module, containing the class socket.
You need to do socket.socket(...) or from socket import socket:
>>> import socket
>>> socket
<module 'socket' from 'C:\Python27\lib\socket.pyc'>
>>> socket.socket
<class 'socket._socketobject'>
>>>
>>> from socket import socket
>>> socket
<class 'socket._socketobject'>
This is what the error message means:
It says module object is not callable, because your code is calling a module object. A module object is the type of thing you get when you import a module. What you were trying to do is to call a class object within the module object that happens to have the same name as the module that contains it.
Here is a way to logically break down this sort of error:
"module object is not callable. Python is telling me my code trying to call something that cannot be called. What is my code trying to call?"
"The code is trying to call on socket. That should be callable! Is the variable socket is what I think it is?`
I should print out what socket is and check print(socket)
Assume that the content of YourClass.py is:
class YourClass:
# ......
If you use:
from YourClassParentDir import YourClass # means YourClass.py
In this way, you will get TypeError: 'module' object is not callable if you then tried to call YourClass().
But, if you use:
from YourClassParentDir.YourClass import YourClass # means Class YourClass
or use YourClass.YourClass(), it works.
Add to the main __init__.py in YourClassParentDir, e.g.:
from .YourClass import YourClass
Then, you will have an instance of your class ready when you import it into another script:
from YourClassParentDir import YourClass
Short answer: You are calling a file/directory as a function instead of real function
Read on:
This kind of error happens when you import module thinking it as function and call it.
So in python module is a .py file. Packages(directories) can also be considered as modules.
Let's say I have a create.py file. In that file I have a function like this:
#inside create.py
def create():
pass
Now, in another code file if I do like this:
#inside main.py file
import create
create() #here create refers to create.py , so create.create() would work here
It gives this error as am calling the create.py file as a function.
so I gotta do this:
from create import create
create() #now it works.
Here is another gotcha, that took me awhile to see even after reading these posts. I was setting up a script to call my python bin scripts. I was getting the module not callable too.
My zig was that I was doing the following:
from mypackage.bin import myscript
...
myscript(...)
when my zag needed to do the following:
from mypackage.bin.myscript import myscript
...
myscript(...)
In summary, double check your package and module nesting.
What I am trying to do is have a scripts directory that does not have the *.py extension, and still have the 'bin' modules to be in mypackage/bin and these have my *.py extension. I am new to packaging, and trying to follow the standards as I am interpreting them. So, I have at the setup root:
setup.py
scripts/
script1
mypackage/
bin/
script1.py
subpackage1/
subpackage_etc/
If this is not compliant with standard, please let me know.
It seems like what you've done is imported the socket module as import socket. Therefore socket is the module. You either need to change that line to self.serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM), as well as every other use of the socket module, or change the import statement to from socket import socket.
Or you've got an import socket after your from socket import *:
>>> from socket import *
>>> serv = socket(AF_INET,SOCK_STREAM)
>>> import socket
>>> serv = socket(AF_INET,SOCK_STREAM)
Traceback (most recent call last):
File "<input>", line 1, in <module>
TypeError: 'module' object is not callable
I know this thread is a year old, but the real problem is in your working directory.
I believe that the working directory is C:\Users\Administrator\Documents\Mibot\oops\. Please check for the file named socket.py in this directory. Once you find it, rename or move it. When you import socket, socket.py from the current directory is used instead of the socket.py from Python's directory. Hope this helped. :)
Note: Never use the file names from Python's directory to save your program's file name; it will conflict with your program(s).
When configuring an console_scripts entrypoint in setup.py I found this issue existed when the endpoint was a module or package rather than a function within the module.
Traceback (most recent call last):
File "/Users/ubuntu/.virtualenvs/virtualenv/bin/mycli", line 11, in <module>
load_entry_point('my-package', 'console_scripts', 'mycli')()
TypeError: 'module' object is not callable
For example
from setuptools import setup
setup (
# ...
entry_points = {
'console_scripts': [mycli=package.module.submodule]
},
# ...
)
Should have been
from setuptools import setup
setup (
# ...
entry_points = {
'console_scripts': [mycli=package.module.submodule:main]
},
# ...
)
So that it would refer to a callable function rather than the module itself. It seems to make no difference if the module has a if __name__ == '__main__': block. This will not make the module callable.
I faced the same problem. then I tried not using
from YourClass import YourClass
I just copied the whole code of YourClass.py and run it on the main code (or current code).it solved the error
you are using the name of a module instead of the name of the class
use
import socket
and then
socket.socket(...)
its a weird thing with the module, but you can also use something like
import socket as sock
and then use
sock.socket(...)
I guess you have overridden the builtin function/variable or something else "module" by setting the global variable "module". just print the module see whats in it.
Here's a possible extra edge case that I stumbled upon and was puzzled by for a while, hope it helps someone:
In some_module/a.py:
def a():
pass
In some_module/b.py:
from . import a
def b():
a()
In some_module/__init__.py:
from .b import b
from .a import a
main.py:
from some_module import b
b()
Then because when main.py loads b, it goes via __init__.py which tries to load b.py before a.py. This means when b.py tries to load a it gets the module rather than the function - meaning you'll get the error message module object is not callable
The solution here is to swap the order in some_module/__init__.py:
from .a import a
from .b import b
Or, if this would create a circular dependency, change your file names to not match the functions, and load directly from the files rather than relying on __init__.py
I got the same error below:
TypeError: 'module' object is not callable
When calling time() to print as shown below:
import time
print(time()) # Here
So, I called time.time() as shown below:
import time
print(time.time()) # Here
Or, I imported time from time as shown below:
from time import time # Here
print(time())
Then, the error was solved:
1671301094.5742612
I had this error when I was trying to use optuna (a library for hyperparameter tuning) with LightGBM. After an hour struggle I realized that I was importing class directly and that was creating an issue.
import lightgbm as lgb
def LGB_Objective(trial):
parameters = {
'objective_type': 'regression',
'max_depth': trial.suggest_int('max_depth', 10, 60),
'boosting': trial.suggest_categorical('boosting', ['gbdt', 'rf', 'dart']),
'data_sample_strategy': 'bagging',
'num_iterations': trial.suggest_int('num_iterations', 50, 250),
'learning_rate': trial.suggest_float('learning_rate', 0.01, 1.0),
'reg_alpha': trial.suggest_float('reg_alpha', 0.01, 1.0),
'reg_lambda': trial.suggest_float('reg_lambda', 0.01, 1.0)
}
'''.....LightGBM model....'''
model_lgb = lgb(**parameters)
model_lgb.fit(x_train, y_train)
y_pred = model_lgb.predict(x_test)
return mse(y_test, y_pred, squared=True)
study_lgb = optuna.create_study(direction='minimize', study_name='lgb_regression')
study_lgb.optimize(LGB_Objective, n_trials=200)
Here, the line model_lgb = lgb(**parameters) was trying to call the cLass itself.
When I digged into the __init__.py file in site_packages folder of LGB installation as below, I identified the module which was fit to me (I was working on regression problem). I therefore imported LGBMRegressor and replaced lgb in my code with LGBMRegressor and it started working.
You can check in your code if you are importing the entire class/directory (by mistake) or the target module within the class.
from lightgbm import LGBMRegressor
def LGB_Objective(trial):
parameters = {
'objective_type': 'regression',
'max_depth': trial.suggest_int('max_depth', 10, 60),
'boosting': trial.suggest_categorical('boosting', ['gbdt', 'rf', 'dart']),
'data_sample_strategy': 'bagging',
'num_iterations': trial.suggest_int('num_iterations', 50, 250),
'learning_rate': trial.suggest_float('learning_rate', 0.01, 1.0),
'reg_alpha': trial.suggest_float('reg_alpha', 0.01, 1.0),
'reg_lambda': trial.suggest_float('reg_lambda', 0.01, 1.0)
}
'''.....LightGBM model....'''
model_lgb = LGBMRegressor(**parameters) #here I've changed lgb to LGBMRegressor
model_lgb.fit(x_train, y_train)
y_pred = model_lgb.predict(x_test)
return mse(y_test, y_pred, squared=True)
study_lgb = optuna.create_study(direction='minimize', study_name='lgb_regression')
study_lgb.optimize(LGB_Objective, n_trials=200)
A simple way to solve this problem is export thePYTHONPATH variable enviroment. For example, for Python 2.6 in Debian/GNU Linux:
export PYTHONPATH=/usr/lib/python2.6`
In other operating systems, you would first find the location of this module or the socket.py file.
check the import statements since a module is not callable.
In Python, everything (including functions, methods, modules, classes etc.) is an object.