After reading some existing answers about this topic, I still can't get my head around how this works.
I have a program that manages imports the following way
from ...src.folder.sub_folder import file_name
For then executing methods with
file_name.method_name()
This works, but the interpreter im using indicates Attempted relative import beyond top-level package so I wanted to instance this files the right way (For learning and also removing the annoying red indicator).
Lets say I have the following tree (This is a snippet of the actual big tree, but the idea stays the same)
PackageTest/
main.py
src/
tasks/
__init__.py
tasks.py
security/
recon/
__init__.py
some_scan.py
With the following contents
tasks.py
def function_1():
print("Function1")
return
def function_2():
print("Function2")
return
def function_3():
print("Function3")
return
How can I reference the methods present in tasks.py from some_scan.py?
Im doing the following with no success.
some_scan.py
import src.tasks as tasks
def call_task():
tasks.function1()
call_task()
Related
I have a folder structure for project with small application and some additional scripts, which are not directly used by application (they are used for some data science on data in data folder):
project
data
doc
src
└───scripts1
script1_1.py
script1_2.py
scripts2
script2_1.py
application
common
__init__.py
config.py
enrichment
__init__.py
module1.py
module2.py
__init__.py
app.py
Script app.py is always entry point and is used for starting the application. I want to be able to use relative imports in this folder structure. For example I want to import Configuration class, which is located in config.py inside of module1.py. But when I run app.py and have this import statement in module1.py:
from ..common.config import Configuration
I get the following error:
File ".../project/src/application/enrichment/module1.py", line 6, in <module>
from ..common.config import Configuration
ValueError: attempted relative import beyond top-level package
I would also need to import enrichment modules in app.py, guess it should work similarly:
from .enrichment.module1 import <func or class>
I have read multiple threads on module importing, but I am still not able to reconstruct the behavior, where I am able to use these relative imports and not get the ValueError. In one older project I used path importing in __init__.py files, but I hope it can be solved somehow better, because it was a bit magic for me. Really thanks for any help.
My package looks something like this:
/mypackage
|-- __init__.py
|-- moduleA.py
|-- moduleB.py
|-- oldModule.py
And __init__.py looks something like this:
from mypackage.moduleA import abc
from mypackage.moduleB import jkl
This allows users to simply code as follows:
import mypackage as mpkg
mpkg.abc()
mpkg.jkl()
By default (using the above import) users have no access to functions in oldModule.py.
This is because 95% of users don't need that stuff (it's old).
For the 5% of users who need functions from the oldModule, they can write:
from mypackage.oldModule import xyz
xyz()
Many of these "oldModule" users don't need access to the functions imported by __init__.py
The problem is, when a user only does from mypackage.oldModule import xyz it is clear the the entire __init__.py is also being run.
Is there any way that I can either
set it up so that __init__.py does not run if user does from mypackage.oldModule import ..., or
can I see the structure of the import statement from inside __init__.py,so that I can detect that the user did from mypackage.oldModule import ...,and then I can use an if block inside __init__.py to decide which parts of the code to run ??
Thanks.
There's no way to do exactly what you're asking for. When you import some_package.some_module, the package always gets fully loaded before the submodule does.
But there might be some ways to mitigate the issues this causes in your case. If you want to avoid importing the moduleA and moduleB submodules when a user is looking for oldModule, you can perhaps make their loading lazy, rather than eager.
Starting in Python 3.7, PEP 562 enables a module to have a function named __getattr__ which will work like a method of the same name defined in a class. When an attribute of the module is looked up but not found, the __getattr__ function will be called with the name.
So in your __init__.py file, you could do:
# from mypackage.moduleA import xyz don't do these imports unconditionally any more
# from mypackage.moduleB import abc
def __getattr__(name):
global xyz, abc
if name == 'xyz':
from .moduleA import xyz
return xyz
elif name == 'abc':
from .moduleB import abc
return abc
raise AttributeError(f"module {__name__} has no attribute {name}")
If you have a lot more names you need to protect this way (not just two), you may want to write more general code that can process a list of names (probably one per sub-module) and import them properly using importlib and writing directly into the package's __dict__. That will be more convenient than trying to use global statements and writing a separate if branch for each name.
I want to import asyncore from a different directory, because I need to make some changes to how asyncore works, and don't want to modify the base file.
I could include it in the folder with my script, but after putting all the modules I need there it ends up getting rather cluttered.
I'm well aware of making a sub directory and putting a blank __init__.py file in it. This doesn't work. I'm not exactly sure what happens, but when I import asyncore from a sub directory, asyncore just plain stops working. Specifically; the connect method doesn't get run at all, even though I'm calling it. Moving asyncore to the main directory and importing it normally removes this problem.
I skimmed down my code significantly, but this still has the same problem:
from Modules import asyncore
from Modules import asynchat
from Modules import socket
class runBot(asynchat.async_chat, object):
def __init__(self):
asynchat.async_chat.__init__(self)
self.connect_to_twitch()
def connect_to_twitch(self):
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect(('irc.chat.twitch.tv',6667))
self.set_terminator('\n')
self.buffer=[]
def collect_incoming_data(self, data):
self.buffer.append(data)
def found_terminator(self):
msg = ''.join(self.buffer)
print(msg)
if __name__ == '__main__':
# Assign bots to channels
bot = runBot()
# Start bots
asyncore.loop(0.001)
I'm sure this is something really simple I'm overlooking, but I'm just not able to figure this out.
Use sys.path.append -- see https://docs.python.org/3/tutorial/modules.html for the details.
Update: Try to put a debug print to the beginning and end of sources of your modules to see whether they are imported as expected. You can also print __file__ attribute for the module/object that you want to use to see, whether you imported what you expected -- like:
import re
#...
print(re.__file__)
I have a (python3) package that has completely different behaviour depending on how it's init()ed (perhaps not the best design, but rewriting is not an option). The module can only be init()ed once, a second time gives an error. I want to test this package (both behaviours) using py.test.
Note: the nature of the package makes the two behaviours mutually exclusive, there is no possible reason to ever want both in a singular program.
I have serveral test_xxx.py modules in my test directory. Each module will init the package in the way in needs (using fixtures). Since py.test starts the python interpreter once, running all test-modules in one py.test run fails.
Monkey-patching the package to allow a second init() is not something I want to do, since there is internal caching etc that might result in unexplained behaviour.
Is it possible to tell py.test to run each test module in a separate python process (thereby not being influenced by inits in another test-module)
Is there a way to reliably reload a package (including all sub-dependencies, etc)?
Is there another solution (I'm thinking of importing and then unimporting the package in a fixture, but this seems excessive)?
To reload a module, try using the reload() from library importlib
Example:
from importlib import reload
import some_lib
#do something
reload(some_lib)
Also, launching each test in a new process is viable, but multiprocessed code is kind of painful to debug.
Example
import some_test
from multiprocessing import Manager, Process
#create new return value holder, in this case a list
manager = Manager()
return_value = manager.list()
#create new process
process = Process(target=some_test.some_function, args=(arg, return_value))
#execute process
process.start()
#finish and return process
process.join()
#you can now use your return value as if it were a normal list,
#as long as it was assigned in your subprocess
Delete all your module imports and also your tests import that also import your modules:
import sys
for key in list(sys.modules.keys()):
if key.startswith("your_package_name") or key.startswith("test"):
del sys.modules[key]
you can use this as a fixture by configuring on your conftest.py file a fixture using the #pytest.fixture decorator.
Once I had similar problem, quite bad design though..
#pytest.fixture()
def module_type1():
mod = importlib.import_module('example')
mod._init(10)
yield mod
del sys.modules['example']
#pytest.fixture()
def module_type2():
mod = importlib.import_module('example')
mod._init(20)
yield mod
del sys.modules['example']
def test1(module_type1)
pass
def test2(module_type2)
pass
The example/init.py had something like this
def _init(val):
if 'sample' in globals():
logger.info(f'example already imported, val{sample}' )
else:
globals()['sample'] = val
logger.info(f'importing example with val : {val}')
output:
importing example with val : 10
importing example with val : 20
No clue as to how complex your package is, but if its just global variables, then this probably helps.
I have the same problem, and found three solutions:
reload(some_lib)
patch SUT, as the imported method is a key and value in SUT, you can patch the
SUT. Example, if you use f2 of m2 in m1, you can patch m1.f2 instead of m2.f2
import module, and use module.function.
I am confused about some behavior of Python. I always thought importing a module basically meant executing it. (Like they say here: Does python execute imports on importation) So I created three simple scripts to test something:
main.py
import config
print(config.a)
config.a += 1
print(config.a)
import test
print(config.a)
config.py
def get_a():
print("get_a is called")
return 1
a = get_a()
test.py
import config
print(config.a)
config.a += 1
The output when running main.py is:
get_a is called
1
2
2
3
Now I am confused because I expected get_a() to be called twice, once from main.py and once from test.py. Can someone please explain why it is not? What if I really wanted to import config a second time, like it was in the beginning with a=1?
(Fortunately, for my project this behavior is exactly what I wanted, because get_a() corresponds to a function, which reads lots of data from a database and of course I only want to read it once, but it should be accessible from multiple modules.)
Because the config module is already loaded so there's no need to 'run' it anymore, just return the loaded instance.
Some standard library modules make use of this, from example random. It creates an object of class Random on first import and reuses it when it gets imported again. A comment on the module reads:
# Create one instance, seeded from current time, and export its methods
# as module-level functions. The functions share state across all uses
#(both in the user's code and in the Python libraries), but that's fine
# for most programs and is easier for the casual user than making them
# instantiate their own Random() instance.