I have been struggling to figure out how to organize my code effectively with many modules of various types. I have a three layered folder system. The parent directory contains the main.py file which imports and runs the main code. The modules are in a subfolder called lib, and different modules are placed in further subfolders. However, when I import one module from a subfolder, that module itself fails to import modules in the same sub directory. Sorry if this is a bad question, but I cannot figure out why the import statement isn't working, and I have looked at google and stack overflow and have not found any similar problem. All advice is welcome.
Parent Directory
---->lib
|--->module_group
|module_one.py(that tries but fails to import module_one)
|module_two.py
|main.py
In main I run
from lib.module_group.module_one import Module_One
Which works until an error is handled saying that there is no such module as module two. However, when I run module_one by itself, it works fine with the following import statement.
from module_two import Module_Two
Set PYTHONPATH to the directory you want to act as your root. For example, if you want to use the following code:
from lib.module_group.module_one import Module_One
Then set PYTHONPATH to the directorying containing lib. For example:
lib/module_group/module_one.py:
from lib.module_group.module_two import Module_Two
lib/module_group/module_two.py:
class Module_Two:
print('Loaded Module_Two')
Then, to run module_one.py directly and still enable it to use lib.module_group.module_two to load Module_Two, use something like:
$ PYTHONPATH="${PWD}:${PYTHONPATH}" python3 lib/module_group/module_one.py
Loaded Module_Two
Related
I was working on a simple project, bike_rentals. I thought all was well until I began writing test files for the project.
I created a new directory and name it TestBikeRental and saved it within the BikeRentals directory, which contained the scripts for my project. I created the file init.py within the BikeRentals and TestBikeRentals directory to make them each be a package. Within the top directory is the script run_script.py that I use to run the modules\scripts contained in the package BikeRentals.
Directory structure
The problem comes up when I try to import the py script db_connection (underlined in yellow), within the Test_bike_rentals.py.
This is how tried to imported db_connection.
Importing db_connection
After numerous and numerous attempts, the errors that I kept receiving were these two:
from BikeRentals.BikeRentals.db_connection import Connect
ModuleNotFoundError: No module named 'BikeRentals'
from..db connection import Connect
ImportError: attempted relative import with no known parent package
To view the sourcecode in github repo:
https://github.com/Brownred/Python-and-SQL
I tried to import a module but got an error after numerous attempts. I was expecting no error when it comes to importing a module.
I'm trying to test my code with Pytest framework, but for some reason I don't succeed importing my own code...
You have to import your code from the top module, like this:
from se2pdf.se2pdf import changeFileExtensionToPDF
the first se2pdf is the folder name and the second your file name. If it doesn't work, make sure there is a _init_.py file in the modules.
I am having trouble with importing python modules from two git repos as the code was not designed as a package, and there are name collisions which disabled me from being able to use sys.path.append in a straight-forward (albeit hacky?) manner.
Note that the problem is not particular of git-repos however I have encountered this problem only in this scenario because of trying to interface with existing work.
The folder structure looks like following:
project
- repo1
- foo.py
- bar.py
- run.py
- repo2
- foo.py
- bar.py
- run.py
test.py
Both foo.py import bar.py with import bar statements, which works fine as long as you are running the run.py scripts in the repo, however if I try to import repo1.foo and repo2.foo I run into multiple issues.
On attempting the following code
# test.py
import repo1.foo
import repo2.foo
I get the error ModuleNotFoundError: No module named 'bar', this makes some sense as nothing in the path makes either bar.py accessible.
However if I try to append paths 'repo1' and 'repo2' then I can only import either repo1.bar or repo2.bar depending on order of appends, which is not an acceptable solution.
As the repositories are not my code I would prefer to not change them, but instead wrap them into a namespaced package if possible, however I am unable to find any solutions for the problem.
TL;DR
Update:: As I was depending on this work extensively I published it as a Python package available here: https://pypi.org/project/packagify/ . You can install it instead of copying the old gist.
Add the class from this gist: https://gist.github.com/rijulg/3ea372bef35adb68e27080c949c942af in your code and you can use it as following
from packagify import Packagify
package = Packagify("/home/workspace/my_package")
object = package.import_module("module", ["object"])
object1, object2 = package.import_module("module", ["object1", "object2"])
Methods Tried
Installing the directory as a module using setuptools, this did not work as on loading the module various parts inside it threw ModuleNotFoundError as they were trying to access siblings without namespaces.
Various importlib functions, there are a few that allow loading python modules from .py files directly. This might have worked if the modules I wanted to import were contained within a file, but then perhaps there would be other solutions as well. As the problem stood I wasn't able to use this method as the module upon loading wasn't able to locate it's siblings.
Adding sys.path as mentioned in the question, but I also tried appending the path, loading module and then removing the path before loading the other module. This also did not work because presumably the modules were loaded into python cache and subsequent loads from different directory were ignored because of name clashes.
Finally, I noticed something weird happening when I changed the import level using the __import__ function and I tried the following snippet which worked fine on my toy example. However this still had various issues that I encountered while trying to load the actual repos I wanted to work with. As the solution appeared to be going in the right direction I made the class linked in the gist mentioned a the beginning of the post.
import builtins
original_import = __import__
def my_import(name, globals, locals, fromlist, level):
print("my_import", name, globals['__package__'], level)
try:
return original_import(name, globals, locals, fromlist, level)
except:
level = level + 1 if globals['__package__'] is not None else level
return original_import(name, globals, locals, fromlist, level)
builtins.__import__ = my_import
from repo1.foo import main as main_a
from repo2.foo import main as main_b
main_a()
main_b()
I have created a simplified version as to focus solely on getting the relative path to work. This is my file structure:
|
-project
|-package1
| |--page
| |-__init__
|
|-package2
|-test
|-__init__
I am trying to import page into test. However, I get the error that package1 is not a module. Below I have typed all that are in my code. Very simple. I am just trying to import page into test. Is there anything I am missing (file or page set up) that is preventing me from importing?
page.py
one='half'
two='ling'
tests.py
import os
import sys
three = (one+two)
print(three)
Have you tried "from package1 import page" into your test.py? Or "from package1.page import page"?
UPDATE
When import something, Python Interpreter search in the following places:
Built-ins
Current Directory
$PYTHONPATH, environment variable
some other directory related to the installation
The last three make up to be sys.path.
In your case, to import package1 into some script in package2, there's 2 ways:
Add project path into PYTHONPATH.
Dynamically append project path into sys.path
I guess you would appreciate the latter solution, just add
import sys
sys.path.append('..')
in front of everything and it will work.
Plus: It's kind of inconvenient to use the module not inside the current directory though. I've seen only a few actual python project, and What I've seen is some of them use a single main.py in the project root to run the whole project, including test-cases. Maybe this structure is more recommended.
Hope it helps~
Original Answer:
This dir structure works fine on my computer with:
import package.page as page
page.foo() # a function in page
Could I have a guess: your current working directory may be not under your project directory.
To check, test about this:
import os
print(os.getcwd())
If the output is not your current directory, that's my case. I used to mess with this before.
To avoid this, you can:
cd to your directory before running Python
run os.chdir(...) in your code, which is to change your working directory.
If not this case, please provide more information.
I have a working python3 program that imports various standard library modules, e.g.
import re
import os
import csv
...
It runs without problem when I execute it directly. Now I want to run it through an exec call from an outside script, like this:
with open("main.py") as main:
exec(main.read())
But I get an error:
ImportError: no module named 'csv'
How can I make the executed script import all modules correctly?
I thought that all these three modules belong to the python standard library so, why the first two modules seem to be found while the third is not?