I am attempting to run timeit.timeit in the following class:
from contextlib import suppress
from pathlib import Path
import subprocess
from timeit import timeit
class BackupVolume():
'''
Backup a file system on a volume using tar
'''
targetFile = "bd.tar.gz"
srcPath = Path("/BulkData")
excludes = ["--exclude=VirtualBox VMs/*", # Exclude all the VM stuff
"--exclude=*.tar*"] # Exclude this tar file
#classmethod
def backupData(cls, targetPath="~"): # pylint: disable=invalid-name
'''
Runs tar to backup the data in /BulkData so we can reorganize that
volume. Deletes any old copy of the backup repository.
Parameters:
:param str targetPath: Where the backup should be created.
'''
# pylint: disable=invalid-name
tarFile\
= Path(Path(targetPath /
cls.targetFile).resolve())
with suppress(FileNotFoundError):
tarFile.unlink()
timeit('subprocess.run(["tar", "-cf", tarFile.as_posix(),'
'cls.excludes[0], cls.excludes[1], cls.srcPath.as_posix()])',
number=1, globals=something)
The problem I have is that inside timeit() it cannot interpret subprocess. I believe that the globals argument to timeit() should help but I have no idea how to specify the module namespace. Can someone show me how?
I think in your case globals = globals() in the timeit call would work.
Explanation
The globals argument specifies a namespace in which to execute the code. Due to your import of the subprocess module (outside the function, even outside the class) you can use globals(). In doing so you have access to a dictionary of the current module, you can find more info in the documentation.
Super simple example
In this example I'll expose 3 different scenarios.
Need to access globals
Need to access locals
Custom namespace
Code to follow the example:
import subprocess
from timeit import timeit
import math
class ExampleClass():
def performance_glob(self):
return timeit("subprocess.run('ls')", number = 1, globals = globals())
def performance_loc(self):
a = 69
b = 42
return timeit("a * b", number = 1, globals = locals())
def performance_mix(self):
a = 69
return timeit("math.sqrt(a)", number = 1, globals = {'math': math, 'a': a})
In performance_glob you are timing something that needs a global import, the module subprocess. If you don't pass the globals namespace you'll get an error message like this NameError: name 'subprocess' is not defined
On the contrary, if you pass globals() to the function that depends on local values performance_loc the needed variables for the timeit execution a and b won't be in the scope. That's why you can use locals()
The last one is a general scenario where you need both the local vars in the function and general imports. If you keep in mind that the parameter globals can be specified as a dictionary, you just need to provide the necessary keys, you can customize it.
Related
I wan to import all the functions and class into a module/file of Python in high level file just passing a variable that contains the low level file name.
I have a application with several module like:
__all__ = ['MyClass1', 'my_function1']
class MyClass1():
pass
def my_function1():
pass
that previous was import at the high level file as:
from sub_module1 import *
from sub_module2 import *
...
# To direct use, of the different subfiles:
obj1 = MyClass1()
obj2 = MyClass2()
The application became a plugin based and I have to dynamic import all module into a folder and provide direct access to all objects defined into __all__ of those submodules.
The code bellow imports fine the submodules but I don not give my direct access to the directives defined into __all__ of those files.
from os import path
from importlib import import_module
directory_name = ## Define the plugins dir.
for importer, package_name, _ in iter_modules([directory_name]):
module_specification = importlib.util.spec_from_file_location(
package_name, path.join(directory_name, package_name + '.py'))
module_loader = importlib.util.module_from_spec(module_specification)
module_specification.loader.exec_module(module_loader)
How do I put those object define into __all__ of the submodules inside locals() of the high module?
I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))
Use the imp module, or the more direct __import__() function.
You can use exec:
exec("import myapp.commands.%s" % command)
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
Please note that I have already referred to StackOverflow question here. I post this question to investigate if calling __post_init__ is safe or not. Please check the question till the end.
Check the below code. In step 3 where we load dataclass A from yaml string. Note that it does not call __post_init__ method.
import dataclasses
import yaml
#dataclasses.dataclass
class A:
a: int = 55
def __post_init__(self):
print("__post_init__ got called", self)
print("\n>>>>>>>>>>>> 1: create dataclass object")
a = A(33)
print(a) # print dataclass
print(dataclasses.fields(a))
print("\n>>>>>>>>>>>> 2: dump to yaml")
s = yaml.dump(a)
print(s) # print yaml repr
print("\n>>>>>>>>>>>> 3: create class from str")
a_ = yaml.load(s)
print(a_) # print dataclass loaded from yaml str
print(dataclasses.fields(a_))
The solution that I see for now is calling __-post_init__ on my own at the end like in below code snippet.
a_.__post_init__()
I am not sure if this is safe recreation of yaml serialized dataclass. Also, it will pose a problem when __post_init__ takes kwargs in case when dataclass fields are dataclasses.InitVar type.
This behavior is working as intended. You are dumping an existing object, so when you load it pyyaml intentionally avoids initializing the object again. The direct attributes of the dumped object will be saved even if they are created in __post_init__ because that function runs prior to being dumped. When you want the side effects that come from __post_init__, like the print statement in your example, you will need to ensure that initialization occurs.
There are few ways to accomplish this. You can use either the metaclass or adding constructor/representer approaches described in pyyaml's documentation. You could also manually alter the dumped string in your example to be ''!!python/object/new:' instead of ''!!python/object:'. If your eventual goal is to have the yaml file generated in a different manner, then this might be a solution.
See below for an update to your code that uses the metaclass approach and calls __post_init__ when loading from the dumped class object. The call to cls(**fields) in from_yaml ensures that the object is initialized. yaml.load uses cls.__new__ to create objects tagged with ''!!python/object:' and then loads the saved attributes into the object manually.
import dataclasses
import yaml
#dataclasses.dataclass
class A(yaml.YAMLObject):
a: int = 55
def __post_init__(self):
print("__post_init__ got called", self)
yaml_tag = '!A'
yaml_loader = yaml.SafeLoader
#classmethod
def from_yaml(cls, loader, node):
fields = loader.construct_mapping(node, deep=True)
return cls(**fields)
print("\n>>>>>>>>>>>> 1: create dataclass object")
a = A(33)
print(a) # print dataclass
print(dataclasses.fields(a))
print("\n>>>>>>>>>>>> 2: dump to yaml")
s = yaml.dump(a)
print(s) # print yaml repr
print("\n>>>>>>>>>>>> 3: create class from str")
a_ = yaml.load(s, Loader=A.yaml_loader)
print(a_) # print dataclass loaded from yaml str
print(dataclasses.fields(a_))
I have a list of modules that should be imported automatically and in a dynanamic way.
Here is a snippet from my code:
for m in modules_to_import:
module_name = dirname(__file__)+ "/" +m
spec = importlib.util.spec_from_file_location("package", module_name)
imported_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(imported_module)
I measured the time and it becomes slower and slower after each import. Is there some solution to this or why does it become slower? Thanks a lot!
I have not timed it but why dont you simplify your code. Looking at your code, you want to import module that are in the same directory as that file. By default, when you import a module, that the first place it look for.
First let's create some files to import all in the same directory:
First.py
def display_first():
print("I'm first")
Second.py
def display_second():
print("I'm second")
Third.py
def display_third():
print("I'm third")
So one way to do it is putting your modules in a dict that you can use afterwards. I'm using here a dict comprehension here to build that dict:
Solution1.py
import importlib
modules_to_import = ["First", "Second", "Third"]
modules_imported = {x: importlib.import_module(x) for x in modules_to_import}
modules_imported["First"].display_first()
modules_imported["Second"].display_second()
modules_imported["Third"].display_third()
Or if you really want to use to use the dotted notation to access a module's content, your could use a named tuple to help:
Solution2.py
import importlib
import collections
modules_to_import = ["First", "Second", "Third"]
modules_imported = collections.namedtuple("imported_modules", modules_to_import)
for next_module in modules_to_import:
setattr(modules_imported, next_module, importlib.import_module(next_module))
modules_imported.First.display_first()
modules_imported.Second.display_second()
modules_imported.Third.display_third()
So I have two different files somewhat like this:
file1.py
from file2 import *
foo = "bar"
test = SomeClass()
file2.py
class SomeClass :
def __init__ (self):
global foo
print foo
However I cannot seem to get file2 to recognize variables from file1 even though its imported into file1 already. It would be extremely helpful if this is possible in some way.
Importing file2 in file1.py makes the global (i.e., module level) names bound in file2 available to following code in file1 -- the only such name is SomeClass. It does not do the reverse: names defined in file1 are not made available to code in file2 when file1 imports file2. This would be the case even if you imported the right way (import file2, as #nate correctly recommends) rather than in the horrible, horrible way you're doing it (if everybody under the Sun forgot the very existence of the construct from ... import *, life would be so much better for everybody).
Apparently you want to make global names defined in file1 available to code in file2 and vice versa. This is known as a "cyclical dependency" and is a terrible idea (in Python, or anywhere else for that matter).
So, rather than showing you the incredibly fragile, often unmaintainable hacks to achieve (some semblance of) a cyclical dependency in Python, I'd much rather discuss the many excellent way in which you can avoid such terrible structure.
For example, you could put global names that need to be available to both modules in a third module (e.g. file3.py, to continue your naming streak;-) and import that third module into each of the other two (import file3 in both file1 and file2, and then use file3.foo etc, that is, qualified names, for the purpose of accessing or setting those global names from either or both of the other modules, not barenames).
Of course, more and more specific help could be offered if you clarified (by editing your Q) exactly why you think you need a cyclical dependency (just one easy prediction: no matter what makes you think you need a cyclical dependency, you're wrong;-).
When you write
from file2 import *
it actually copies the names defined in file2 into the namespace of file1. So if you reassign those names in file1, by writing
foo = "bar"
for example, it will only make that change in file1, not file2. Note that if you were to change an attribute of foo, say by doing
foo.blah = "bar"
then that change would be reflected in file2, because you are modifying the existing object referred to by the name foo, not replacing it with a new object.
You can get the effect you want by doing this in file1.py:
import file2
file2.foo = "bar"
test = SomeClass()
(note that you should delete from foo import *) although I would suggest thinking carefully about whether you really need to do this. It's not very common that changing one module's variables from within another module is really justified.
from file2 import * is making copies. You want to do this:
import file2
print file2.foo
print file2.SomeClass()
global is a bit of a misnomer in Python, module_namespace would be more descriptive.
The fully qualified name of foo is file1.foo and the global statement is best shunned as there are usually better ways to accomplish what you want to do. (I can't tell what you want to do from your toy example.)
After searching, I got this clue: https://instructobit.com/tutorial/108/How-to-share-global-variables-between-files-in-Python
the key is: turn on the function to call the variabel that set to global if a function activated.
then import the variabel again from that file.
i give you the hard example so you can understood:
file chromy.py
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def opennormal():
global driver
options = Options()
driver = webdriver.Chrome(chrome_options=options)
def gotourl(str):
url = str
driver.get(url)
file tester.py
from chromy import * #this command call all function in chromy.py, but the 'driver' variable in opennormal function is not exists yet. run: dir() to check what you call.
opennormal() #this command activate the driver variable to global, but remember, at the first import you not import it
#then do this, this is the key to solve:
from chromy import driver #run dir() to check what you call and compare with the first dir() result.
#because you already re-import the global that you need, you can use it now
url = 'https://www.google.com'
gotourl(url)
That's the way you call the global variable that you set in a function. cheers
don't forget to give credit
while I do the test following the idea of #robertspierre to put all global variables in a glv.py file and then import it in other files where it is used, the demo codes is given bellow, hope it helps:
the global variable file, glv.py:
# glv.py
glvB = True
glvA = 100
glvS = "tiger"
glvList = [1, 2, 3]
glvTuple = (1, "a")
glvDict = {"Name": "tiger", "age": 100}
sub1.py, it's a file that will import the glv.py file. Two functions are defined to show and change the global variable data in glv.py, showData() and changeData(),
# sub1.py
import glv
def showData():
print(f"*****glv in sub1*****\n"
f"glvB={glv.glvB}\n"
f"glvA={glv.glvA}\n"
f"glvS={glv.glvS}\n"
f"glvList={glv.glvList}\n"
f"glvTuple={glv.glvTuple}\n"
f"glvDict={glv.glvDict}\n")
def changeData():
glv.glvB = False
glv.glvA = 200
glv.glvS = "bactone"
glv.glvList = [4, 5, 6]
glv.glvTuple = (2, "b")
glv.glvDict = {"Name": "bactone", "age": 0}
sub2.py is another file:
# sub2.py
import glv
def showData():
print(f"*****glv in sub2*****\n"
f"glvB={glv.glvB}\n"
f"glvA={glv.glvA}\n"
f"glvS={glv.glvS}\n"
f"glvList={glv.glvList}\n"
f"glvTuple={glv.glvTuple}\n"
f"glvDict={glv.glvDict}\n")
def changeData():
glv.glvB = True
glv.glvA = 300
glv.glvS = "bactone"
glv.glvList = [7, 8, 9]
glv.glvTuple = (3, "c")
glv.glvDict = {"Name": "bactone1", "age": 10}
finally we test the global variable in main.py:
import glv
import sub1
import sub2
def showData():
print(f"*****initial global variable values*****\n"
f"glvB={glv.glvB}\n"
f"glvA={glv.glvA}\n"
f"glvS={glv.glvS}\n"
f"glvList={glv.glvList}\n"
f"glvTuple={glv.glvTuple}\n"
f"glvDict={glv.glvDict}\n")
if __name__ == "__main__":
showData() # show initial global variable
sub1.showData() # show global variable in sub1
sub1.changeData() # change global variable in sub1
sub2.showData() # show global variable in sub2
sub2.changeData() # change global variable in sub2
sub1.showData() # show global variable in sub1 again
the results turns out to be:
*****initial global variable values*****
glvB=True
glvA=100
glvS=tiger
glvList=[1, 2, 3]
glvTuple=(1, 'a')
glvDict={'Name': 'tiger', 'age': 100}
*****glv in sub1*****
glvB=True
glvA=100
glvS=tiger
glvList=[1, 2, 3]
glvTuple=(1, 'a')
glvDict={'Name': 'tiger', 'age': 100}
*****glv in sub2*****
glvB=False
glvA=200
glvS=bactone
glvList=[4, 5, 6]
glvTuple=(2, 'b')
glvDict={'Name': 'bactone', 'age': 0}
*****glv in sub1*****
glvB=True
glvA=300
glvS=bactone
glvList=[7, 8, 9]
glvTuple=(3, 'c')
glvDict={'Name': 'bactone1', 'age': 10}
we can see all kinds of data type works and the change of global variable is automatically reloaded.
I came to the conclusion that you can import globals, but you can not change them once imported. The only exception is if you pass them as arguments. I would love to be wrong on this, so let me know if there is a way to effectively re import updated globals. The two codes below will run.
from b import * # import all from b.py
global alpha # declare globals
global bravo
global charlie
alpha = 10 # assign values to globals
bravo = 20
charlie = 15
def run_one():
one(alpha) # pass the global to b.py
def run_two():
two() # rely on import statement in b.py
def run_three():
global charlie # declare the global to change it
charlie = 40 # change the value for charlie
print("charlie:", charlie, " --> global value changed in a.py run_three()")
def run_three_again(): # print charlie again from b.py
three()
def run_four(): # re import charlie in b.py
four()
if __name__ == "__main__": # prevent the code from being executed when b.py imports a.py
run_one() # run through all the functions in a.py
run_two()
run_three()
run_three_again()
run_four()
Also:
from a import * # import all from a.py
def one(alpha):
print("alpha: ", alpha, " --> global passed as argument in one()")
def two():
print("bravo: ", bravo, " --> global imported from a.py in two()")
def three():
print("charlie:", charlie, " --> global imported from a.py in three() but is not changed")
def four():
from a import charlie # re import charlie from a.py
print("charlie:", charlie, " --> global re-imported in four() but does not change")
The output from the print statements are below:
alpha: 10 --> global passed as argument in one()
bravo: 20 --> global imported from a.py in two()
charlie: 40 --> global value changed in a.py run_three()
charlie: 15 --> global imported from a.py in three() but is not changed
charlie: 15 --> global re-imported in four() but does not change
All given answers are wrong. It is impossible to globalise a variable inside a function in a separate file.
Just put your globals in the file you are importing.